Introduction

Patient reported outcome measures (PROMS) provide information on patient perceptions of their own health and wellbeing, including in response to interventions such as elective surgery [1,2,3]. By engaging patients in co-evaluation of their responses to surgery, health professionals and administrators can implement patient-informed, co-produced policies, procedures and interventions [4, 5]. Prior to large scale implementation of PROMS across hospitals globally, it is important to understand PROM feasibility, acceptability and outcomes from a surgical patient perspective [6, 7].

Feasibility refers to the ease of treatment implementation, practicality, integration, demand and acceptability [8]. In the context of receiving a PROM survey, each of the constructs are key. For example, the burden of a survey refers to the effort required to participate at different time points, coherence is the extent to which the patient understands the survey, and the self-efficacy refers to patient confidence that they can complete the survey correctly [9]. Acceptability from the patient perspective is particularly important for PROM surveys, as patient response rate, a key outcome of PROM implementation, acts as a surrogate for acceptability [10]. Acceptability is described as the extent to which the people who provide or receive an intervention view it as appropriate, effective and helpful [9]. It also incorporates elements of burden, coherence, ethics, opportunity cost and self-efficacy [9].

This current PROM study for elective surgery patients is a part of a larger research program designed to develop Australian ePROM implementation recommendations, called ‘AusPROM’, with a detailed protocol previously published [11]. In summary, the AusPROM research program contains a number of phased studies that embed patient and staff co-design into the implementation process, through an iterative process, and this included the identification of barriers and facilitators from the patient and staff perspective.

In the current study we sampled patients having a broad range of elective surgeries from four major Australian hospitals, aiming to evaluate (1) technical feasibility of the technology used to implement the PROMs across geographically dispersed sites, (2) response rates for automated short message service (SMS) versus email survey delivery, (3) the degree of recovery at one and four weeks post-surgery (Quality of Recovery 15 Item PROM; QoR-15) [12, 13], and (4) patient acceptability based on survey and focus group results. The patient surveys and focus group were designed to enable patient co-design in the development of the broader AusPROM recommendations, through the development of a set of patient-driven recommendations.

Methods

This study has been reported according to the CONSORT statement: extension to randomised pilot and feasibility trials [14] in “Appendix 1”. As noted in the aims, the current study reports on four distinct areas from this broader research program including technical feasibility, response rate, degree of recovery and patient acceptability.

Study design

We used a mixed-methods design to test the technical feasibility, response rate, degree of recovery and patient acceptability of electronic PROMS.

Patient co-design

To improve acceptability, usability and uptake, patient feedback and co-design were embedded throughout the study [4]. This included a patient co-designing and co-authoring the project from its concept (VR); patients completing acceptability questions alongside the PROM survey; as well as a patient focus group to further explore PROM survey acceptability and patient driven recommendations for ongoing PROM implementation [4, 5].

Participants

Survey participants were patients who had elective surgery in four hospitals in Australia. Inclusion criteria were adults aged 18+, having either elective day-surgery or elective surgery requiring an overnight hospital admission. Exclusion criteria included a pregnancy related procedure or an investigative procedure (see “Appendix 2” for full list of excluded procedures). Consent was via an opt-in consent tick-box prior to commencing the survey. The pre-test surveys were distributed through one hospital in New South Wales, Australia. The pilot site surveys were distributed through four of the health services 29 hospitals which provide elective surgery. The four pilot sites were selected based on a sample of convenience and were located in New South Wales, Queensland and Victoria, Australia. Three of the four hospitals had an emergency department and all four were located in metropolitan areas.

Across the health service, there are hospital specific consumer groups. The consumers are generally past patients from the health service. The patient focus group participants were members of the hospital specific consumer groups (i.e., not current patients). Individuals were approached through the Quality Manager, based on a request from the Corporate Consumer Consultant (Chairperson of the Healthscope National Consumer Advisory Council). The sample of convenience aimed to obtain a mix of past patients across the different states and territories of Australia. Inclusion criteria were adults 18+ who were members of the hospital specific consumer groups. There were no exclusion criteria. Written informed consent was required prior to participation in the focus group.

Intervention and study instrument

The intervention was the administration of the PROM survey, based on the QoR-15 tool, the week prior to surgery, and one and four weeks post-surgery. The QoR-15 was derived from the Quality of Recovery 40 item (QoR-40) tool [12]. It has 15 items each rated on a 11-point scale from 0 to 10, with a maximum score of 150 The QoR-15 has reported good validity, reliability and responsiveness and is brief to administer (< 3 min) [12, 15]. The QoR-15 can be used pre-surgery, 24 h post-surgery, as well as from weeks to months post-surgery, as a measure of change over time [13, 16, 17], supporting this studies use of the QoR-15 the week prior to surgery, and one and four weeks post-surgery. The minimal clinical important difference of the QoR-15 is 8.0 [16].

With permission from the QoR-15 author [13], questions 7 and 8 of the QoR-15 tool were modified for the pre-surgery surveys as they were designed for the post-surgery period. Questions 7 was modified from “Getting support from hospital doctors and nurses” to “Getting support from Health Professionals”, and Question 8 was modified from “Able to return to work or usual home activities” to “Able to participate in work or usual home activities”.

The QoR-15 tool was chosen due to its valid use pre and post-surgery, the short completion time and its applicability across most surgery types [13, 18,19,20], enabling the health service to implement one consistent and inclusive tool. In addition, the individual items within the tool can be used to isolate the more difficult areas of recovery post-surgery, and enable the health service to target these areas to improve care and recovery.

Outcomes

  1. (i)

    Technical feasibility of the technology used to implement the PROMs across geographically dispersed sites

Outputs and outcomes for the PROM pre-pilot surveys included development of agreed list of surgical procedures to be excluded from the survey distribution list, as well as observations of the differences between the pre-surgery and the post-surgery survey distribution lists.

  1. (ii)

    Response rates for automated short message service (SMS) versus email survey delivery

Response rate for SMS and email survey invitations were reported separately and combined for the pre-surgery, one week post-surgery and four weeks post-surgery surveys. In addition, at the four sites, separate to the PROM survey, was a long-standing Patient Experience survey. As a part of PROM feasibility testing, we examined if the introduction of the PROM survey impacted the response rate of the Patient Experience survey. To test this, the Patient Experience response rates were reviewed across the four sites during the PROM study.

  1. (iii)

    Degree of recovery at one and four weeks post-surgery as measured by the QoR-15

The PROM survey, focussed on the QoR-15, was administered electronically pre-surgery, one week post-surgery and four weeks post-surgery. Patients were independently invited to participate in a survey at each time point. This was because someone included in the pre-surgery survey may have been excluded in the post-surgery survey if the planned surgery changed and was one of the excluded surgery types.

  1. (iv)

    Patient acceptability based on survey and focus group results (quantitative and qualitative acceptability questions)

Outcomes for patient acceptability, via survey questions and the patient focus group, include a survey question “Based on the different aspects of acceptability which are important to you, how do you rate the acceptability of the survey just completed?” measuring of acceptability based on a 0–10 Likert scale (0 = Not acceptable, 10 = Highly acceptable), and an open-ended question asking “Can you please note which aspects of acceptability are important to you? And, how we could modify the survey to be more acceptable to you (or if it is ok just the way it is)?”. In addition, there was a focus group for further qualitative data on acceptability.

The patient survey and focus group results were used to help develop patient driven recommendations for ongoing PROM implementation. It is noted that this study only reports on the patient perspective and that similar work was completed for the staff perspective and this has been reported separately.

The theoretical framework chosen to support the selection and implementation of PROMs was the “PROM cycle” [21]. The PROM cycle has four main phases [21]. The first is “goal” setting, to determine the objective for PROM implementation; the second phase is “selection” and testing of the appropriate PROM; the third identifies the “indicator” which includes the steps of defining and testing the quality indicator; and the fourth is “use” and involves the steps of implementing and evaluating the PROM [21]. Feasibility studies such as the current investigation are vitally important during phases two and three of the PROM-cycle, to test the PROM and confirm quality indicators to show successful implementation.

Sample size

The PROM pre-pilot surveys aimed to invite up to 100 participants, the survey implementation at four pilot sites aimed to invite up to 2700 participants, and both were based on a sample of convenience. Based on a previous national survey response rate across the same heath service of 37% [22], the PROM survey response rate was also expected to be between 30–40%. The patient focus group aimed for up to 10 participants, and was conducted after the survey results were analysed to ensure the themes which emerged in the survey, were explored in the focus group.

Statistical analysis

Results from the PROM pre-pilot surveys are reported descriptively. Results for the PROM survey implementation at four pilot sites are reported as a number and percentage for the response rate, and as a mean and standard deviation for the QoR-15 scores pre and post-surgery. Pre-surgery QoR-15 scores were compared to post-surgery scores using independent t-tests reporting the mean difference and confidence interval. A chi-square statistic was used to determine if there was a difference between proportion within age groups, proportion female, and proportion of day surgery versus overnight surgery between pre-surgery, one week post-surgery and four week post-surgery groups. Responses are reported separately for day surgery, overnight surgery and for combined day/overnight surgery, as a number of the surgeons perform surgery which requires both day and overnight admissions, and this reporting structure will ensure aggregate results are meaningful to the surgeons. Statistical significance was assumed at p < 0.05 and SPSS [23] was used for the analysis.

Qualitative results for patient acceptability, via survey questions and the patient focus group, were analysed using a content analysis and then themed according to the seven constructs of the theoretical framework of acceptability (TFA) [9]. The constructs were based on affective attitude: surgical patient feelings about the survey; burden: the effort required by the patient to participate; and perceived effectiveness: perception by the patient that the survey is likely to achieve its purpose. In addition there was ethicality: the surveys “fit” with individual value systems; intervention coherence: extent to which the patient understands the survey; opportunity costs: extent to which benefits or values of the patient must be given up to engage in the survey; and self-efficacy: patients confidence that they can complete the survey correctly [9]. These themed qualitative results were then critically reviewed by the research team to derive feasibility and acceptability recommendations from the patient perspective.

Results

  1. (i)

    Technical feasibility of the technology used to implement the PROMs across geographically dispersed sites

In April 2021, the pre-pilot survey used email to invite 80 patients to participate over the pre-pilot period. While all 80 received the pre-surgery survey, only 67 received the post-surgery surveys. Patient survey distribution lists included for the pre-surgery survey and the post-surgery survey differed because of late additions or cancellations prior to surgery, and intra-operative changes to the planned surgery, for example a patients would be excluded from the pre-surgery PROM survey distribution list if they had an investigative procedure planned, however, if this extended to an interventional procedure, they would be added to the post-surgery survey distribution list. A list of surgical procedures which were excluded has been detailed in “Appendix 2”.

The pre-surgery survey had 26 responses (response rate 32.5%, n = 26/80). The one and four weeks post-surgery surveys each had 15 responses (response rate 22.4%, n = 15/67). The technical feasibility testing on the pre-pilot survey resulted in a robust party automated/party manual process being developed for the identification of the pre-surgery PROM survey distribution list for PROM survey implementation at four pilot sites, compared to the fully automated post-surgery PROM survey distribution list.

  1. (ii)

    Response rates for automated short message service (SMS) versus email survey delivery

Between April 2021 and July 2021, there were 5985 surveys responses (response rate 29.8%, n = 5985/20,052) and these were distributed across the pre-surgery survey (response rate 44.5%, n = 1756/3944), the one-week post-surgery survey (response rate 33.3%, n = 2682/8054) and the four weeks post-surgery survey (response rate 19.2%, n = 1547/8054).

Two sites used SMS and two sites used email to invite patients to participate. The response rate for SMS was 34.8% (n = 3108/8919) and the median time to complete the survey was 2 min and 10 s and the response rate for email was 25.8% (n = 2877/11,133) and the median time to complete the survey was 2 min and 29 s (Table 1).

Table 1 Response rate
Full size table

At the four sites, separate to the PROM survey, was a long standing Patient Experience survey. As a part of PROM feasibility testing, we examined if the introduction of the PROM survey impacted the response rate of the Patient Experience survey. To test this, the Patient Experience response rates were reviewed and these remained consistent across the four sites during the PROM study with a response rate ranging from 29.9 to 31.5% prior to introducing the PROM survey (October to December 202), and a rate ranging from 29.4 to 31.7% during PROM survey introduction (April to June 2021; Table 2).

Table 2 PROM survey impact on the previously implemented Patient Experience questionnaire
Full size table

While patient characteristics were similar for those who completed the pre-surgery survey, compared to the one week post-surgery survey, compared to the four weeks post-surgery, based on a chi-square statistic the proportion of patients in each of the age brackets did significantly differ between the groups (p < 0.01), however the difference between proportion female, and proportion of day surgery versus overnight surgery, did not significantly differ (p > 0.05), (Table 3).

  1. (iii)

    Degree of recovery at one and four weeks post-surgery as measured by the Quality of Recovery 15 Item PROM (QoR-15)

Table 3 Patient characteristics for the pre-surgery survey, and the one and four weeks post-surgery surveys
Full size table

Across the four pilot sites there were 489 different surgeons who had patients complete a PROM survey, with each surgeon having an average of 3.50 (SD 5.68) patients complete the pre-surgery survey, 6.44 (SD 7.54) patients complete the 1 week post-surgery survey and 4.32 (SD 4.50) patients complete the 4 weeks post-surgery survey.

For combined overnight and day admissions, prior to surgery, the mean QoR-15 score was 122.69 (SD 25.23; n = 1021), one week post-surgery it was 113.08 (SD 27.74; n = 1906) and 4 weeks post-surgery it was 123.39 (SD 26.84; n = 1051). While there was a significant difference in the score from pre-surgery to one week following surgery, and again from one and four weeks post-surgery, there was no significant difference between QoR-15 scores pre-surgery and four weeks post-surgery. At one-week post-surgery, 825 of the 1,906 responses (43%) exceeded 122.6 (pre-surgery average), and at four weeks post-surgery, 676 of the 1,051 responses (64%) exceeded 122.6 (pre-surgery average). When this was separated into the subgroups of day admissions and overnight admissions, the findings were similar (Table 4; Fig. 1).

  1. (iv)

    Patient acceptability based on survey and focus group results (quantitative and qualitative acceptability questions)

Table 4 QoR-15 scores
Full size table
Fig. 1
figure 1

Overall PROM response on the QoR-15 prior to surgery and one and four weeks following surgery

Full size image

During the first, second and third surveys, there were 3739 responses to the survey question rating acceptability, 76% (n = 2830/3739) rated 8/10 or above for acceptability where 10/10 indicated highly acceptable (Fig. 2).

Fig. 2
figure 2

Patient rating for PROM survey acceptability

Full size image

In the PROM survey, 1877 (31%) patients provided a breadth of additional comments regarding patient acceptability and 4108 (69%) did not. Comments were spread across the pre-surgery surgery (n = 507, 27%), the one week post-surgery survey (n = 863, 46%) and the four week post-surgery survey (n = 507, 27%). The single patient focus group (n = 8 participants) had representation from NT, SA, NSW, ACT and Victoria, and explored patient acceptability of introducing a PROM survey (focus group interview guide; “Appendix 3”).

COVID-19 response during pilot data collection

It is noted that the COVID-19 pandemic had an impact on the study during the final weeks of data collection. This involved stopping the pre-surgery survey at two sites (New South Wales) two weeks earlier than planned (completing 10 weeks of the 12 weeks of planned data collection), due to the manual steps associated with the pre-surgery survey to remove participants with excluded surgery types. This was defined as a project “Red Phase”—refer to “Appendix 4” for details related to the “COVID-19 contingency plan”. The one week and four weeks post-surgery surveys were not impacted.

Feasibility and acceptability patient recommendations

To provide structure, the patient recommendations have been presented under the 4-Phases of the PROM cycle as well as the 7-domains of patient acceptability for receiving a health service, in the following Text Box.

Discusssion

This authorship team has completed a research program which resulted in the development of Australian ePROM implementation recommendations, called ‘AusPROM’ [11, 24]. The current study represents the consumer voice and brings patient co-design into the AusPROM recommendation development process. Patients articulated how they felt about completing a PROM survey before and after elective surgery, and co-designed a set of “patient driven” recommendations on how to roll out the survey. These “patient driven” recommendations provided innovative and practical solutions regarding how the survey could be rolled out. These recommendations have been a key driver in a Delphi technique that was used to confirm the final set of AusPROM recommendations [24].

The current study found that implementing PROMS electronically for elective surgery hospital patients was technically feasible and did not require additional infrastructure. In agreement with a 2022 systematic review of global studies by Sokas [25] hospital patients found participating in the ePROM acceptable before and after their surgery, and achieved a similar response rate to other long term hospital surveys. Based on the general direction of the results, for this elective surgery group of patients the PROM discriminated change in the patients surgical recovery over time, with a full recovery for most patients by four-weeks.

Despite being conducted in the height of the COVID-19 pandemic, this comprehensive analysis had almost 6,000 survey responses. It showed favourable responses for implementing PROMS, which is consistent with PROMS implementation in other diagnostic categories such oncology [26] and total knee replacement [27], as well as across broader patient populations [25, 28]. Feasibility testing revealed minor and infrequent technical difficulties in automated SMS or email administration of the PROM prior to surgery. The QoR-15 response rate was higher for SMS compared to email. From the week prior to surgery to the week immediately after elective surgery, patients experienced reduction in this patient-centred outcome and this was reversed by four weeks post-surgery. From the patient perspective, the PROM survey was highly acceptable with 76% of patients rating 8/10 or above for acceptability.

When considering all online devices, such as smart phones and computers, email invitations and SMS invitations to survey participation have a similar response rate, however, when only considering the smart phone response rate, the SMS yields a higher response rate [29, 30]. This finding was consistent for the current study where the SMS response rate (36%) exceeded the email response rate (25%).

The average quality of recovery score in the current study, measured through the QoR-15 tool, prior to elective surgery was 123 out of a possible 150 (n = 1021). This pre-surgery score was consistent with other patient cohorts undergoing mixed surgery types with similar pre-surgery scores of 123 (n = 127) [12] and 125 (n = 363) [31], yet higher than a patient cohort undergoing hip replacement with a score of 114 (n = 115) [32]. Post-operatively, the average quality of recovery score in the current study dipped to 113 (n = 1906) the week after surgery then returned to 123 (n = 1051) 4 weeks after surgery, indicating that based on the general direction of the results, the PROM discriminated change in the patients surgical recovery over time, consistent with the literature [12, 31]. When the cohort was separated into the subgroups of day admissions and overnight admissions, the findings were similar. It is possible that this data set does not represent the final status of recovery after surgery, as further gains may have been achieved after the 4 week post-surgery time point.

Patient acceptability of the current PROM survey was high, based on quantitative and qualitative acceptability questions within the survey and a patient focus group. In comparison, a small number of systematic reviews have concluded that there is either some evidence that PROMs are acceptable to the patients affected by cancer or cystic fibrosis [33, 34], or that there is a paucity in the literature to conclude that PROMs are (or are not) acceptable to the patients affected by chronic fatigue syndrome, a hip fracture or kidney disease [35,36,37].

It has been described as wrong to seek patient feedback in healthcare and then not use this information to influence clinical practice [38]. From the onset, the current study intended to use patient feedback to guide innovative and practical solutions to shape co-design. The themed qualitative results, which developed the 14 feasibility and acceptability recommendations from the patient perspective, will provide a patient voice to the next stage of this research program where the final “AusPROM” Recommendations [11, 24] will be developed. Patient recommendations included defined roles and responsibilities for the nursing, medical, administrative and management staff in relation to the PROM survey process. However, it is unknown if these views would have differed prior to the impact on staff demands due to the COVID-19 pandemic.

Limitations to this study included an end point of 4 week post-surgery which may not have reported the final status of recovery after surgery, excluding any non-electronic modes of survey distribution, as well as the impact of COVID-19 during the final weeks of data collection. The study was conducted in Australia and it is not known whether similar results would be obtained in other regions of the world or for different cultures or case-mixes, or in health services with different technology systems.

Conclusion

Implementing PROMS electronically for elective surgery hospital patients was feasible, acceptable and showed changes in outcome over time. Patient involvement facilitated innovative and practical solutions to implementation and the formulation of recommendations.