Background

Incomplete, unclear or poor quality reporting of patient-reported outcome (PRO) endpoints in randomised controlled trials (RCTs) is a well-documented problem [1, 2], which can lead to research waste in two key ways. Firstly, unclear reporting may lead to inaccurate messages, misinterpretation or misunderstanding of the aims, methods, findings, or relevance of the data. In turn, this reduces the ability of policy makers, funders, clinicians and patients to use the PRO data for clinical and policy decision-making [3,4,5]. Secondly, poorly communicated PRO research may lead others to assume that certain research questions have not been answered [4]. This could lead others to repeatedly or unnecessarily reproduce the research and spend additional, scarce research funding and time to answer the same questions., [3]. Similarly, failing to report certain findings will most certainly lead others to believe that research questions have not been answered. This applies to trials for which PROs have not been reported at all, or where certain measured PRO domains from multi-dimensional measures have not been reported. Non-reporting is particularly concerning, given that past studies have estimated that PROs are left completely unreported for 38–80% trials[6, 7]. In one of these reviews, PRO data were left unreported for 49,568 participants across 61 trials and multiple time points [6], which is unethical, as it devalues the participation of patients in clinical trials and prevents valuable data from being used to inform future patient care.

Reporting guidelines are simple, structured tools for health researchers to use while writing manuscripts. They provide a minimum list of information needed to ensure a manuscript can be understood by readers, used for clinical decision-making, included in systematic reviews, and to ensure the research is reproducible [5, 8]. Evidence of sub-optimal PRO reporting [1, 2, 9, 10] led to the 2013 release of the CONsolidated Standards Of Reporting Trials (CONSORT) PRO extension [11, 12] reporting guidelines. CONSORT-PRO aims to promote transparent reporting of RCTs in which PROs are primary or important secondary outcomes. If used appropriately it can help reduce waste due to incomplete or unclear PRO reporting, and it can best achieve this goal if used in its entirety. Its use is endorsed by many stakeholders, including Enhancing the QUAlity and Transparency of health Research (EQUATOR) [13] and the European Medicine’s Agency [14]. The CONSORT-PRO Extension includes 14 items: five PRO-specific extension items and nine PRO-specific elaborations to CONSORT-2010 items. Addressing all 14 items promotes complete reporting of clinical trial PRO endpoints. Whilst adherence to the CONSORT-PRO Extension does not imply the study was designed and conducted according to highest research standards, adherence will allow such value judgements to be made by the reader.

Since the 2013 publication of CONSORT-PRO, several studies have reviewed PRO research publications to assess the extent to which these reports address the CONSORT-PRO criteria (e.g., [6, 15,16,17]). However, the appraisal checklists used in these review articles have varied. Some have used a 14-item checklist (one item per assessment point), while others have split certain CONSORT-PRO items into multiple items, acknowledging that certain checklist items address multiple reporting requirements. Some reviews have limited their appraisals to only the five PRO-specific extension items. Not including the nine PRO-specific CONSORT-2010 elaborations when assessing completeness of PRO reporting may cause readers to misinterpret what constitutes comprehensive reporting and could perpetuate poor reporting practices. In turn, incomplete reporting can lead to waste as described above, and may have implications for knowledge translation. Knowledge translation refers to the “synthesis, exchange, and application of knowledge by relevant stakeholders to accelerate the benefits of global and local innovation in strengthening health systems and improving people’s health” [18] p.9.

Although it must be noted that the CONSORT-PRO guidance was never intended to be used as an evaluation tool, it continues to be used in this way. Review studies to date have either explicitly stated or implied that a high score indicates high-quality reporting. However, if CONSORT-PRO items are left out of such evaluations, there is a risk that readers may be misled about how to write their future PRO articles completely and transparently—which is a knowledge translation concern.

We aimed to appraise the use of the CONSORT-PRO Extension in “review articles” that have assessed the quality of reporting in PRO research publications and to describe the reporting of PRO research across reviews. We also discuss how this impacts knowledge translation and potential misuse of the CONSORT-PRO Extension.

Methods

Our review methodology followed PRISMA guidelines [19].

Search strategy

Medline, Pubmed and CINAHL were systematically searched from 2013 (year of CONSORT-PRO publication) until 25 March 2021 using the following terms: ((((((((CONSORT-PRO) OR ((CONSORT adj3 patient-reported outcome))) OR ((CONSORT adj3 PRO))) OR ((reporting quality adj3 PRO))) OR ((reporting quality adj3 patient-reported outcome*))) OR ((reporting complet* adj3 patient-reported outcome*))) OR ((reporting complet* adj3 PRO))) OR (reporting quality adj3 PROs)) OR (reporting complet* adj3 PROs). These databases were chosen for their relevance and because key papers known to the authors were indexed in these databases. Duplicate references were removed in Endnote X9.

Selection criteria

Articles reviewing the completeness of reporting of PRO endpoints in RCTs using CONSORT-PRO criteria were included. Articles that reviewed the completeness of PRO reporting using other checklists were excluded. Abstracts were double screened using these criteria by two authors (RMB, OLA). Full text articles were reviewed by one author (RMB) and checked against criteria by OLA. All discrepancies were resolved through discussion.

Data extraction

Two authors (RMB, OLA) extracted the data described in Box 1 from included articles. Data were summarised using descriptive statistics and narrative synthesis.

Results

Our database searches retrieved 152 articles, of which 14 met criteria for inclusion in our review (Fig. 1), listed in Appendix 1. The 14 review articles were published between 2015 and 2020, including a total of 1186 trials published between 1998 and 2019. The most common disease area studied by the review articles was oncology, n = 8 (Table 1). Two review articles also reviewed protocols of the included studies using SPIRIT-PRO.

Fig. 1
figure 1

PRISMA flow diagram

Full size image
Table 1 Characteristics of the 14 included review studies
Full size table

Variation in use of CONSORT-PRO as an evaluation tool

Although all studies used the CONSORT-PRO checklist to evaluate reporting of PROs, the definition of what was being evaluated in each review differed (Table 1). Definitions of what was assessed included “adherence to CONSORT-PRO criteria” n = 5 [16, 17, 21,22,23], “reporting quality” n = 3 [15, 24, 25], “methodological quality” n = 2 [26, 27], “reporting completeness” n = 2 [6, 28], “PRO reporting” n = 1 [29], and “reporting standards” n = 1 [30]. Many of the articles used multiple different terms throughout the manuscript.

Some studies modified CONSORT-PRO criteria for evaluation, and sometimes these changes were not described or justified in the methods, but apparent from the presentation of results or supplementary material. Only six studies used the full CONSORT-PRO checklist without modification or with minimal justified modifications [6, 17, 21, 23, 25, 29]. The remaining eight studies made significant or unjustified adjustments to the CONSORT-PRO Extension: n = 5 studies excluded three or more CONSORT-PRO elaboration items without justification, n = 3 studies excluded all elaboration items and assessed studies using only the five CONSORT-PRO Extension items.

Two issues impacted comparability of total scores across these review articles: the CONSORT-PRO items evaluated and their weighting in the total score. While most CONSORT-PRO items were awarded a score of 1 each time it was addressed by each RCTs, some CONSORT-PRO items that contained multiple criteria were divided for scoring, as shown in Table 2. For example, item P6a: “evidence of PRO instrument validity and reliability should be provided or cited if available, including the person completing the PRO and methods of data collection (paper, telephone, electronic, other)” was spilt in to two or three criteria for two and eight studies, respectively. When this was done, the overall value of the item was increased. When this was not done, it was not abundantly clear if studies had addressed all three components of the item, or only part of the criteria.

Table 2 Adherence to CONSORT-PRO criteria- summary of findings from 14 systematic reviews
Full size table

Review articles varied in whether they reported CONSORT-PRO item-level scores only [16, 22, 24, 27, 30], or both total scores (i.e., total number of CONSORT-PRO items addressed) and item-level scores [6, 15, 17, 21, 23, 25, 26, 28, 29]. The highest possible total score varied between review articles (range 11–15), even among those that used the full CONSORT-PRO checklist, due to differences in how individual items were weighted. Of the three studies using just the five CONSORT-PRO Extension items, none reported a total score [16, 22, 27].

Item-level evaluations using the CONSORT-PRO guidelines

The frequency of reporting of some CONSORT-PRO items varied markedly among review articles (Tables 2, 3). For example, identification of the PRO as a primary or secondary endpoint was addressed by more than 60% of RCTs in five reviews [16, 21, 25, 27, 28], but less than 30% of RCTs in three reviews [15, 22, 24]. Statistical approaches to dealing with missing PRO data were poorly reported in RCTs included in all 14 review articles. Across the 14 review articles, the total number of CONSORT-PRO items (as included in each respective evaluation checklist) addressed by RCTs ranged from 0 to 100%.

Table 3 Summary of findings of the 14 review articles
Full size table

One study assessed predictors of higher adherence to CONSORT-PRO items, and determined that citing CONSORT-PRO, publishing in a ‘journal endorsing CONSORT-PRO’ and ‘publishing PROs in a dedicated PRO paper’ were predictors of higher CONSORT-PRO adherence scores [17]. Other studies also noted a trend for higher adherence scores for articles that reported PROs in a dedicated, secondary publication [15, 26, 28].

Journal endorsement of CONSORT-PRO

Of the 13 journals that the 14 included review articles were published in, none recommended use of the CONSORT-PRO Extension specifically, five recommended use of EQUATOR guidelines or appropriate CONSORT guidelines, and nine journals did not mention either in their instructions to authors.

Discussion

Our review found that evaluation checklists developed based on the CONSORT-PRO recommendations are inconsistent and lead to differing evaluations of reporting of RCTs. The key features of the CONSORT-PRO recommendations that have been adapted for these evaluations include: the number of CONSORT-PRO Extension items included, the number of CONSORT-PRO Elaboration items included, the weighting of specific items—particularly those that address multiple aspects of reporting, the wording of certain items and the highest possible total score that could be obtained from evaluation.

As noted earlier, these review articles are, by their nature, sending a message to readers about how PROs should be reported. Readers of these review articles may incorrectly assume that if a CONSORT-PRO item is missing from the evaluation criteria of a specific review, then it is not essential to include that item in future reports. This issue is made even more problematic when articles make judgements about what constitutes a “good” total CONSORT-PRO reporting score. One article, with a highest possible total CONSORT-PRO adherence score of 11, suggested that studies scoring between 6 and 11/11 had high adherence, studies scoring between 3 and 5/11 had medium adherence, and between 1 and 2/11 had low adherence to CONSORT-PRO standards—without accounting for CONSORT-PRO recommendations for articles with a primary PRO endpoint only. As with any reporting standards endorsed by the EQUATOR network, in the interests of promoting high-quality, transparent reporting and reducing waste, all CONSORT-PRO items are recommended for inclusion in publications of PRO trial endpoints, unless specific conditions are specified for individual items. For example, CONSORT-PRO recommends a PRO-specific sample size calculation only for studies with a primary PRO endpoint, so a specific calculation for PROs is not required where PROs are secondary endpoints. The aim of these resources is not to criticise researchers, but to help researchers to publish high-quality and clear reports of their research, so that stakeholders may interpret the findings accurately and use the results appropriately in clinical practice or policy decisions. The implication of not reporting research clearly or completely is that readers may misunderstand the purpose and value of the research, the methods and/or findings. Readers may assume that PRO data does not exist for particular important clinical questions. This may limit results from impacting clinical practice, or result in duplication of research efforts in the future (researchers may assume those research questions need answering in future studies), which may be viewed as a waste of research resources and participant time.

We noted differences in reporting practices between reviews in the CONSORT-PRO items most- and least-often reported. All 14 review articles reported that the statistical approach used for handling missing PRO data (item P12a) was poorly reported, including articles published before and after the CONSORT-PRO was released. This item, in combination with item E16 (number of participants included in each analysis), is of particular importance to help readers judge the potential bias resulting from missing PRO data, and the extent to which results can be generalised to the broader patient population. Numerous publications over the past 30 years have highlighted this reporting issue and offered suggestions for how to address it [31,32,33,34,35,36,37,38,39,40,41,42].

A related issue highlighted by many of the review articles was that many studies did not report results for each assessed domain and assessment time point (E17a). This item was omitted from the evaluation of five of the 14 review articles. In PRO research, multi-domain instruments are often used, yet often only the overall quality of life domain is reported. This may contribute to research waste because readers may assume that other domains were not collected and that research questions around these domains remain unanswered. Given that leading regulators are advocating for the assessment of specific and clearly defined PRO domains that are more proximal to the studied disease or condition [43], there is a clear need for improved identification and definition of PRO endpoints, in addition to specification of their primary or secondary endpoint status. Calvert et al. [11] noted the importance of full reporting of all domains assessed to avoid bias in reporting, in justifying the need for and purpose of this item in the CONSORT-PRO checklist. Options to overcome challenges associated with restrictive manuscript word limits include publishing PROs in a dedicated publication, or including detailed PRO results in a manuscript appendix.

We also noted differences in the recommendations between review articles, which in some cases contradicted one another. The original CONSORT-PRO article [11] recommends that key PRO endpoints are published in the main trial publication, however review studies conducted since have observed that PRO reporting is more complete when PROs are published in a dedicated publication [15, 17, 26, 28]. Accordingly, these reviews have recommended that future trials publish their PRO data in a separate publication as soon as possible after publishing their primary trial endpoints. Two key details to this approach being successful, and to minimise the potential for research waste, are to clearly report trial identifying details (e.g., trial registration number, trial name) across all trial publications (cross-referencing where possible), and to ensure all endpoints are published in a timely manner, as there is a risk that certain endpoints will be left unpublished if not included in a single publication.

Our results highlight the need for training across health and medical setting in all aspects of PRO reporting, to encourage effective use of CONSORT-PRO, with the outcome of reducing research waste. A new initiative, patient-reported outcomes tools: engaging users and stakeholders (PROTEUS) aims to advance the use of PROs in research studies and clinical practice by implementing and disseminating CONSORT-PRO and related resources. The PROTEUS website includes checklists, web tutorials, and other resources to support the optimal use of PROs so that patients, clinicians, and other decision-makers have the information they need to support patient-centered care [44]. Education about both the importance of high-quality research design, conduct and reporting, and how to implement these through use of available resources, is crucial to minimise the risk of PRO research waste in the future.

Recommendations

As noted earlier, CONSORT-PRO was not intended to be used as an evaluation tool, however at least 14 studies have used the guidance in this way since its publication. Whilst these studies have helped to highlight what areas of PRO reporting are in need of improvement generally (despite the existence of guidance), studies using incomplete or modified checklists may also have confused readers about what is recommended in a publication of PRO trial endpoints, which may in turn result in incomplete or poor quality reporting and consequential research waste. We recommend that future studies using the CONSORT-PRO guidance to evaluate studies take care to include all CONSORT-PRO items (elaborations and extensions). We acknowledge that it is difficult to assess multi-component outcomes of the CONSORT-PRO checklist, and it is worth taking a consistent approach to future evaluations by adapting the CONSORT-PRO extension for review purposes so that all components are clearly assessable. We have attached a template (Appendix 2) that addresses all component items, which future reviews may wish to implement. This checklist has been used successfully in three of the included review studies [17, 23, 25].

Future review articles should also encourage readers to seek high-quality PRO advice and support when planning and publishing PRO studies, to ensure that reporting is clear, accurate and in line with recommended standards. This practice would facilitate appropriate knowledge translation of the CONSORT-PRO extension. The PROTEUS website may be useful for this purpose. It covers all aspects of PRO design, methodology, analysis, interpretation and reporting, and draws on expert, internationally endorsed PRO guidance in its content. Research manuscripts are a key source of information for policy makers, funders, clinicians, patients and their families, when making important treatment and clinical decisions, therefore we should strive to produce and communicate data of the highest quality. Glasziou and colleagues have outlined strategies to reduce research waste resulting from poor reporting which apply to PROs, including following EQUATOR reporting guidelines; registering trials, using CrossRef to link reports to trials; making use of archiving software to facilitate data sharing, data back up and collaboration; producing commentaries post-publication; and ensuring findings are presented within research context [3].

Strengths and limitations

We used a robust search strategy across multiple databases to identify suitable manuscripts for this review, and two experienced reviewers double screened all abstracts. We also followed the PRISMA guidance in designing, conducting and reporting our review. However, it is possible that human errors may have led to relevant articles being missed. Our review only identified papers written in English. It is possible that if we had searched international databases, relevant non-English articles may have been identified, particularly as CONSORT-PRO has recently been translated into Japanese [45]. We also acknowledge that it is only fair to describe, not evaluate, RCT publications published before CONSORT-PRO was released in 2013 using CONSORT-PRO, however all reviews included in this review of reviews, included at least one paper published before CONSORT-PRO was released. We included all review studies that met our inclusion criteria and purposefully did not assess their methodological quality. We wanted to focus on the specific CONSORT-PRO-inspired reporting evaluation checklists used, and how these may impact future reporting, rather than the methodological quality of the systematic reviews themselves, as arguably the review checklists used would impact knowledge translation more directly. The completeness of the CONSORT-PRO-inspired checklist is most relevant to our study’s aim, and in the absence of any measures of effect, an additional assessment of methodological quality is unlikely to impact our descriptive findings [46]. As noted above, we found that some evaluation checklists were incomplete, therefore, our summary of item-level reporting of CONSORT-PRO criteria are limited accordingly.

We did not check for overlap between the 1181 studies evaluated across the 14 included reviews. There is potential for overlap, particularly as eight of the reviews were specific to oncology, and two of the eight studies (N = 681 studies) included any cancer diagnosis. The remaining six reviews of oncology RCTs looked at reporting in specific cancer populations. Some of these studies did not publish lists of included studies, therefore we could not check the extent of overlap. Cochrane’s guidance on conducting reviews of reviews suggests that some degree of overlap in descriptive reviews such as ours is not problematic [46].

Three of the reviews included a small number of studies which were not RCTs [23, 24, 28], for which the CONSORT-PRO guidance may be less relevant.

Finally, we wish to reiterate that adherence to reporting guidelines represents only one strategy to reduce research waste and promote research impact. There is no measurable, linear relationship between adherence to the CONSORT-PRO checklist and the extent of research waste, as research impact is a complicated and multi-faceted concept. A poorly designed and conducted study can indeed report every item on the CONSORT-PRO checklist, and a study that is designed and conducted perfectly may not address all items on the CONSORT-PRO checklist in its publication. A paper that adheres to reporting guidelines better-places a reader to assess its design and conduct and to interpret its findings accurately—consequently improving the potential of the research to be impactful and meaningful to clinical practice, and to contribute to systematic reviews or meta-analyses.

Conclusions

Our study has two major findings. Firstly, many PRO studies published after 2013 (when CONSORT-PRO was published) did not report items recommended by CONSORT-PRO, which may lead to research waste if results are consequently misinterpreted or misunderstood. Additional knowledge translation efforts are needed to promote effective use of the CONSORT-PRO guidance. Secondly, studies reviewing PRO publications have omitted or largely modified recommended items from their evaluations, which may cause confusion and further issues for readers regarding how best to report their PRO research in line with the CONSORT-PRO extension.

When reported effectively, PRO data from RCTs can provide valuable information to inform clinical care, regulatory decision-making and health policy. The CONSORT-PRO extension aims to improve the completeness  and transparency of reporting, to maximise research impact and help minimise research waste. Articles reviewing PRO reporting have successfully identified areas in need of improvement, but may also have caused readers to believe that certain CONSORT-PRO items are non-essential or optional due to omitting these items from their evaluations, which is not the case. Future review articles should include all 14 CONSORT-PRO Extensions and Elaborations to avoid confusion or issues with knowledge translation, and should recommend readers access helpful resources to promote high-quality PRO reporting.