Introduction

The direct input from cancer patients and survivors on their needs, symptoms, and experiences of care is fundamental to the delivery of high quality, personalised cancer care [1, 2]. A standardised way to obtain such input is through the collection of patient-reported outcomes (PROs) that provide a ‘status report’ derived directly from patients [3]. PROs are typically collected using patient reported outcome measures (PROMs), which are questionnaires, ideally validated, across a range of health conditions and settings and used to assess overall general health, quality of life, psychological, emotional, and physical well-being [4, 5]. Collection of PROs has been extensively utilised in research, health service performance monitoring, and more recently, in direct cancer care.

Collection of PROs

Benefits of PRO collection in clinical cancer care have been well established and include greater patient satisfaction with care [6], reduced emergency presentations [7, 8] and improved survival [7, 9]. The use of PROs has also been shown to support patient-centred care through increased patient activation [3], predict treatment outcomes and complications [10, 11], and assist in monitoring of the quality and safety with health services [12].

Despite these many benefits, the use of routine PROs in clinical cancer care remains limited [13, 14].To facilitate adoption, health services need explicit recommendations regarding effective implementation of PRO collection systems.

The International Society for Quality of Life Research (ISOQOL) recommended minimum standards for the implementation of PROs used in patient-centred outcomes [15,16,17]. These recommendations stated that PRO initiatives were more likely to be successful if integrated into clinical workflow through direct engagement with all stakeholders, including patients, who would be involved in PRO collection [15,16,17]. A case-series paper further discussed the importance of using implementation science through a multi-level approach [17]. It found that while identified barriers to the implementation of routine collection of PROs were consistent across different health settings, including cancer, enablers were context specific and further work was necessary to develop practical steps for the design of implementation strategies.

While this case-series paper provided a starting point from which to consider PRO implementation more broadly, there remains a need for practical recommendations that use implementation science theory and the real-world perspective of stakeholders. To address this need, the present study aimed to develop priority recommendations for the implementation of PROs into routine clinical cancer care at the health service level. This work first considered the level of importance of implementation factors and then their relative priority resulting in recommendations for implementation that a cancer service should consider when attempting to implement the collection of PROs.

Methods

The study employed a modified Delphi methodology—involving a series of iterative questionnaires to determine relative priorities for implementation [18,19,20]. The Delphi process included four characteristic features: anonymity, iteration with controlled feedback, statistical group response, and expert input [18,19,20].

Participant recruitment

Participant recruitment included those potentially engaged with PRO implementation (e.g. healthcare professionals [patient facing roles in which cancer care is provided], information technology (IT) staff, hospital administrators, and researchers, as well as cancer survivors). A multidisciplinary working group (WG) of experts in PROs, and cancer patients and survivors, had been established by the Clinical Oncology Society of Australia (COSA) to advocate for and promote the use of PROs in cancer, and was invited to provide input as to the design of the study. Following a COSA think tank on PRO use in Australia in 2018, a working group of interested clinicians, researchers, and consumers was formed through an open invitation to members and other interested individuals, to further advance the PRO implementation agenda in Australia. The group published a ‘call to action’ in the Medical Journal of Australia, which advocated for the advancement of personalised cancer care through the collection of PROs [1]. The need for the Delphi study on implementation was identified as the next step. For this Delphi study, participants were recruited by distributing our survey through the clinical and consumer networks of the WG. WG members were sent an invitation email (including the participant information sheet and the link to the study consent form) to complete the Delphi survey, and this was further distributed across their networks using a snowball approach. This WG has wide representation of key stakeholders in cancer survivorship clinical care as members are representatives of a variety of care cancer units/organisations (e.g. tertiary care centres, community care centres) and patient advocate groups across Australia.

Within the survey link, participants were asked to provide consent, basic demographic information, and a contact email. These contact emails were used for round 2 survey distribution, as only participants who engaged in round 1 were invited to participate in round 2. No incentives were provided to facilitate further engagement between rounds. However, two reminder emails were sent to participants of round 1 to complete round 2.

Generation of priority statements

The development of statements used in the Delphi process was informed by a literature review examining implementation of PROs in cancer care settings, the Knowledge to Action (KTA) implementation framework [21], and feedback from the WG.

The study team reviewed literature through keyword searches in PubMed and Medline for articles addressing the implementation of PROs in cancer care settings (Appendix A). The drafting of priority statements occurred iteratively with the literature review. Articles were identified as being relevant if they included either considerations of barriers and/or enablers for the implementation of PROs or discussed points of consideration when deciding how best to implement PROs into clinical practice (e.g. position statements on PRO implementation). Statements were first drafted as close to verbatim from the derived literature. Next, the KTA was utilised as a framework through which to categorise emerging priorities into domains. The KTA is an implementation process model that describes practical steps or stages in translating research into practice, with core concepts of knowledge and creation [21]. The framework outlines six domains for implementation consideration: (1) adapt knowledge to local context, (2) address barriers, (3) develop implementation strategies, (4) monitor use context/adaptations & designing tests of change, (5) evaluate outcomes, and (6) focus on sustainability.

To consolidate the list of statements prior to the first round of the Delphi study, items under each of the KTA domains were reviewed, refined, discussed, and examined for clarity and understanding by the WG.

Priority review process

Drafted statements underwent a two-round modified Delphi process. The online consensus process was facilitated through a survey-based platform (RedCap V10.0.1).

An invitation email (Appendix B) including the participant information sheet (Appendix C) and link to the round 1 Delphi survey was sent to the WG for dissemination. This was distributed by the WG using a nomination and snowball sampling approach. Only participants who engaged in round 1 were invited to participate in round 2.

Round 1—rating of importance

In round 1, participants were asked to rate the importance of each item for the implementation of routine collection of PROs at the health service level using a 9-point Grading of Recommendations Assessment, Development and Evaluation (GRADE) scale [22]. Scores of 7–9 indicated the item was of ‘critical importance’, 4–6 indicated ‘importance’, and 1–3 indicated ‘limited importance’. Free text fields were provided for each survey section encouraging participants to comment on or explain their ratings, and to add or rate new items that they felt were not already covered. The round 1 survey is available as Appendix D. At the conclusion of round 1, the WG was convened to discuss the results and to provide feedback on which items should progress to ranking in round 2.

Round 2—priority ranking

In the second round, participants were presented with the results of round 1 and asked to rank the presented statements in order of importance with ‘1’ being the highest priority item to consider when attempting to implement PROs at the health service level. Free-text fields were provided to suggest additions and/or comments. Round 2 survey is available as Appendix E.

Priority review meeting

The WG convened a final priority review meeting to discuss study results and the associated priority ratings of each item and to discuss generalisability and implications for policy and future research.

Statistical analysis

The study team synthesised the agreement levels of each statement from round 1 through descriptive statistics of agreement levels per item. An additional, conventional content analysis of the free-text fields was undertaken to capture recommended modifications or adjustments to presented statements [23]. Statements that received a ranking of 7–9 from ≥70% of participants and a ranking of 1–3 by ≤ 15% of participants were deemed to have high priority levels [22]. Otherwise, the statement was considered to need re-drafting using either qualitative feedback obtained or WG input. Summary reports of all items included in the Delphi round 1 were produced and disseminated at the start of round 2 to all survey participants for reference and consideration.

Upon completion of Delphi round 2, median ranking scores and interquartile ranges were calculated for each statement. Further assessment of free-text field comments through content analysis was completed and presented where relevant to complement the interpretation of the quantitative statistical results.

This study was approved by the Cancer Council NSW Ethics Committee (ref. #324).

Results

Priority statement generation

Our targeted review search identified one position statement on how to best integrate PROs into clinical practice [24], several scoping/systematic/pooled reviews that discussed implementation of PROs [6, 25,26,27,28,29,30,31], and five methodological and research articles that discussed using implementation approaches to implement and evaluate PRO use in clinical practice [5, 9, 32,33,34]. These sources led to the development of 71 suggested statements (Appendix F).

Statements were organised conceptually into five domains of the KTA framework, combining domains four and five ( [4] monitor use context/adaptations & designing tests of Change and [5] evaluate outcomes) as few statements arose in these domains and those that did were deemed to be complementary to each other. Through WG input and consolidation, 47 statements were next formatted into questionnaire items that formed the first round of the Delphi study.

Priority review process

A total of 70 PRO stakeholders participated in round 1, with 45 continuing to round 2. Table 1 summarises the characteristics of participants. The majority of participants in round 1 were healthcare professionals (51.4%, n=36/70), predominantly medical oncologists (30.8%), treating adult patients (41.0%). Round 2 encapsulated a similar distribution of participants.

Table 1 Characteristics of Delphi participants across both rounds
Full size table

The majority of participants identified their familiarity with PROs as either ‘expert’ (41.4%, n=29/70) or ‘intermediate’ (37.1%, n=26/70) users. Of these, the majority labelled themselves as PRO collectors or analysts (47.4%), followed by PRO collection ‘system builders’ or involved in the design of infrastructure to routinely collect PROs (29.5%), and participants in PRO collection (7.7%).

Participants were from five of eight Australian states and territories with the majority residing in New South Wales and Western Australia (31.4%, n=22/70 each).

Round 1—rating of importance

After round 1, seven statements did not meet the agreed threshold for ‘high priority’ rating (range 62–69%, statements denoted with an asterisk in Table 2). However, as the process of the modified Delphi was not to exclude items that had not met general consensus, but rather to identify the emerging priority rankings of the statements, all statements were retained for round 2. Participants provided a total of 29 free-text field comments, the majority of which (n=22) were general statements of agreement with the presented items or highlighted the item’s importance. A few comments (n=7) suggested reasons for disagreement with items relating to clinician prompting for PRO collection. No new items were recommended nor modifications to existing items suggested.

Table 2 Results from round 1 (importance) and round 2 (ranking) across both Delphi rounds by domain of Knowledge to Action framework
Full size table

Round 2—priority ranking

Round 2 rankings revealed the relative importance of statements; the top five statements for each section of the KTA are summarised in Table 3.

Table 3 Top 5 most highly prioritised statements within each Knowledge To Action domain to consider when attempting to implement PROs into routine clinical cancer care practice
Full size table

The majority of free-text comments described difficulties that participants had ranking the importance of the presented statements (n=10/15), as they considered them to be of similar importance. As such, these participants suggested that their ranking process was instead a reflection of preferred sequential order when implementing PROs at the health service level.

Priority review meeting

The final priority review meeting, attended by 10 of the 15 members of the WG, resulted in unanimous agreement on the top five statements within each domain being of the highest priority for a health service to consider first when attempting to implement a process for routine collection of PROs (Table 3). The WG discussed intentions for future national advocacy for the incorporation of these findings into the national cancer plan.

Discussion

This is the first study that reports on what multidisciplinary users, including cancer patients, health care providers, administrators, and IT professionals, consider as priority actions for implementation of PROs into clinical cancer care.

Based on this work, the key practical step for PRO implementation is for health services to identify current staff capabilities and the services required to undertake the collection of PROs. Next, health services should strive to form strong stakeholder partnerships with those who will be involved with PRO collection and use, especially cancer survivors. Our recommendations further discuss which barriers were seen as priorities to address, with mapping workflow and process and the need for clear articulation as to the evidence behind the value of PRO collection deemed as highest priority. The most important considerations in the development of implementation strategies were to identify support champions and to practically design the integration of PROs in care pathways and clinical workflows. Considered from the perspective of stakeholders, using PROs as quality assurance tools was deemed as an acceptable method of monitoring and evaluating outcomes of PRO implementation. Finally, using routinely collected PRO data to guide and improve clinical care at the health service level was viewed as the most critical priority to ensure sustainability of PRO collection.

The priorities identified in the present study are consistent with those identified by others, notably, the Canadian position statement [24]; however, the Canadian recommendations were derived from health professionals or academic researchers and were not ranked in terms of prioritisation of which items to consider first.

The present study extends the recommendations of the ISOQOL case-series reports [13, 17] by ranking the relevant implementation strategies necessary for PROs implementation from the perspective of the multidisciplinary stakeholders involved.

While previous literature on PRO implementation discussed the barriers and enablers of implementation [6, 15, 26,27,28,29,30,31], the present study provides a new lens through which to consider those barriers and enablers in terms of their relative importance to PRO stakeholders.

The majority of presented items were rated highly during round 1, and the prioritisation rankings in round 2 did not elicit much variation. This lack of outlying variation within our results may be due to our use of the KTA framework. Using this tool allowed for full consideration of all factors relating to implementation, but it may have led to naturally high ratings and low variation within ranking as the domains themselves are evidenced-based. As such, content within the domains would fundamentally be deemed important or of high priority from the outset.

Additionally, it is notable that most of our study’s recommendations focused on the initial set-up and adoption with less attention to sustainability. This may reflect the current state of the field and limited long-term use of PROs in clinical care. Literature suggests that sustainability in PRO collection would be best placed for success if the processes are designed through collaborative efforts with key stakeholders, especially cancer survivors [35, 36] and indeed our ranking, placed particular importance on stakeholder engagement and collaboration at the outset, although did not explicitly connect this to the sustainability outcomes.

Strengths and limitations

Our study strengths lie within the representation of a variety of stakeholder perspectives including cancer survivors. We note however that our sample was overly representative of healthcare professionals, with a limited practical perspective from PRO system builders (e.g. IT specialists). While more specific data on the type of categorisation of healthcare professionals was not collected (e.g. medical doctor, nurse, physiotherapist, psychologist, etc.), our study objective of developing priority recommendations for the implementation of PROs into routine clinical cancer care at the health service level required a general healthcare professional perspective, irrespective of these sub-categorisations. Additionally, the attrition of participants from round 1 to round 2 may have impacted our round 2 results as participants were asked to rank statements in order of importance rather than their general agreement of importance (as they had in round 1). Higher participation numbers in round 2 may have produced a different ranking for statements depending on the personal attributes of the sample; however, this loss to follow-up in round 2 produced no major changes in terms of proportional representation amongst the categories of role groups.

Furthermore, this work was undertaken with the Australian context in mind and facilitated through Australian-based participants and thus may not be generalisable to all health settings or contexts with very different health care systems to the Australian model. While our results were contextually derived in the Australian setting, we believe they have international relevance as the KTA framework used to organise and consider implementation factors is applicable globally [21] and the similarity of our findings to other international recommendations attests to their global relevance.

Despite the limitations, this study stands as the first of its kind in formulating stakeholder-derived recommendations and a practical guide to implementing PROs in the cancer care clinical context. This work can be viewed as a building block from which health services can guide their considered efforts for PRO implementation while considering their local context.

Conclusion

This stakeholder consultation process has identified key priorities in PRO implementation into clinical cancer care that include clinical relevance, stakeholder engagement, communication, and integration within the existing processes and capabilities. Future research should focus on pragmatic application of these recommendations to ensure growing uptake of PROs into clinical practice.