Raising the profile of pilot and feasibility studies in relation to the development, evaluation and implementation of patient-reported outcome measures
This editorial introduces a new special series on the pilot and feasibility testing of patient-reported outcome measures (PROMs) in the on-line open access journal Pilot and Feasibility Studies. Pilot and feasibility studies are typically implemented to address issues of uncertainty before undertaking a larger definitive study such as a randomised controlled trial or large scale survey. This editorial considers the role that such pilot and feasibility testing plays in relation to the development, evaluation and implementation of PROMs. This is often an essential element of PROM research but is typically overlooked—especially within current methodological guidance, reporting space and also debate. This editorial aims to open up a dialogue about the role of pilot and feasibility testing in relation to PROMs. It highlights some of the areas in PROMs research where these types of studies have been carried out and discusses the ways in which the PROM community may be better supported and encouraged to integrate this element of the research process into their PROM-based work.
The application of social science methods in the evaluation of medical care has continued to grow in importance. In particular, there is an increasing demand for the design, development and implementation of questionnaires that can assess patients’ experiences of health and illness. These questionnaires are typically referred to as patient-reported outcome measures (PROMs) or patient-reported outcomes (PROs). They are questionnaires designed to provide a means of measuring the impact of illness, and its associated treatments, or other types of intervention, from the patient’s perspective.
Traditionally, medical care was evaluated using clinical measures of outcome, i.e. measures of mortality and other clinical diagnostic criteria which concentrated upon the physical components of health and ignored the dimensions of well-being and functioning, which could have an impact upon the health status of the patient . However, in the latter half of the twentieth century, there was increasing awareness that health and illness are not purely dependent upon physical well-being. In 1954, the World Health Organization (WHO) emphasised this point in their definition of health as “a state of complete physical mental and social well-being and not merely the absence of disease or infirmity” . Also, in the past, the evaluation of patients’ experiences of health and illness was primarily based upon the objective judgements of clinicians. It has been suggested that these judgements were often based upon intuition and personal experience . However, research has shown that clinical and other such proxy reports made on behalf of a patient (e.g. from parents or carers) are far from objective and show variations and low levels of agreement to those of the patient [4, 5, 6]. As a result, there has been a growing demand to assess and evaluate the other dimensions of well-being which can impact upon the health of patients and to develop measurement tools in the form of questionnaires which can evaluate systematically this subjective impact on well-being beyond the traditional measures of outcome such as mortality or morbidity .
Types and uses of PROMs
There are a large number of PROMs available, and they may differ in their measurement properties, content length and intended purpose. However, typically, they can be categorised as being generic, disease or condition specific. These may also have utility (preference) values estimated for the responses and therefore become preference-based measures. These are used for calculating quality-adjusted life years (QALYs), allowing for the economic value of interventions to be assessed . More recently, with the increasing drive to capture data as part of routine care and find cost-effective, time efficient ways of routinely capturing the impact of treatments and illnesses from the patient’s perspective, there has been a rapid rise in ePROMs and a move away from the traditional mode of paper completions to electronic/web-based solutions. This is evident across a large and diverse range of medical disciplines  including pre-operative assessment , surgery  and cancer  to name a few.
PROMs have an important role in evaluative research as a measure of outcome. This particularly applies to the interpretation of RCT data where their use can provide additional information on the benefits of medical therapies or interventions, as an aid to clinical decision-making . A second related factor is their use in quality assurance and audit . Thirdly, they can assess the health care needs of populations by being used in surveys to capture information on the health needs of populations beyond the traditional mortality and socio-demographic data which are not specific enough to inform decision makers about the allocation of resources .
Most importantly, though, as PROMs are all concerned with providing information on the things which matter most to the patient, they can also provide valuable information to the clinician or other health care professional about patients’ progress. This can aid in the clinical management of the patient by enabling physicians to monitor patients’ progress and consequently influence decisions about treatment. Finally, an important use which is frequently overlooked is that completing a PROM also provides the opportunity for the patient to express the impact of the illness upon his/her well-being. For example, when PROMs have been administered in routine clinical practice, it has been found that patients appreciated the opportunity to report how they were feeling and to be involved with their care .
The use of PROMs in pilot and feasibility studies
Thus, within medicine and the related disciplines, PROMs clearly have a number of important and useful roles and their use is becoming more widespread—particularly with the drive to incorporate these more routinely into clinical practice and medical decision-making [17, 18, 19]. For example, since 2009, the NHS has made it a requirement to collect PROM data from patients before and after surgery in four surgical conditions: hip replacement, knee replacement, varicose veins and groin hernia, with it recently reported that there were plans to extend the PROMs programme over a wider range of long-term conditions and treatments in the NHS, including mental health, chronic obstructive pulmonary disease (COPD), diabetes, epilepsy, heart failure and stroke .
Guidelines for conducting pilot and feasibility studies have been published by both the Medical Research Council (MRC) and National Institute for Health Research (NIHR) , with the MRC reporting that pilot and feasibility testing are interchangeable concepts covering all aspects of preparatory work in their guidelines on complex interventions (www.mrc.ac.uk/complexinterventionsguidance). The pilot and feasibility testing of PROMs often plays an essential role. For example, in terms of the evaluation and implementation of the DH PROM programme the measures were only selected by the Department of Health following testing in numerous pilot studies  based upon a process which involved “piloting their use and reviewing their potential to be rolled out across the NHS” . Similar projects have also been or are currently underway to evaluate the feasibility of implementing PROMs into current practice. For example, one initiative is “The cardiac revascularisation PROMs pilot” . This was originally commissioned by the Department of Health in 2011 but later passed to NHS England in 2013. Patients across 11 English NHS hospital trusts who were on a waiting list for cardiac revascularisation to treat their heart disease were invited to participate in the PROM pilot. The aim of this was to evaluate if it was possible for the NHS to collect good enough data pre- and post treatment. A few other similar feasibility studies have been published which have particularly focused on determining the feasibility of implementing PROM/s and estimating response rates—although other important aspects such as gaining stakeholder and service user feedback, exploring responsiveness and estimating costs associated with the PROM completions were also some key aims of the pilot stage [23, 24, 25].
all items and procedures in a pilot test of whether patients understand the items and instructions included in the PRO instrument .
The FDA recommends cognitive interviewing as one such way of carrying out this assessment and undertaking other small pilot studies to test the face validity of the measure (e.g. that response options and recall periods are appropriately comprehended and that the instrument’s readability is adequate for the intended population). However, more guidance on what the nature of these pilot and feasibility studies may entail is not provided. Similar gaps are evident in other key user manuals and checklists within the PROM field, e.g. COSMIN [27, 28]. It is unclear in these international guidelines how pilot and feasibility studies may be used to support the methodological quality of studies that are focused on PROM development, evaluation and implementation, or if the same recommendations as they currently stand in these documents translate to the pilot and feasibility testing of PROMs also.
Despite this, it appears as though researchers are using their own initiatives to incorporate pilot and feasibility testing. By no means an exhaustive list, some examples of these initiatives include (i) developing questionnaires to determine the relevance and acceptability of PROMs during aspects of face validity ; (ii) testing search strategies in systematic reviews of PROM measures and literature ; (iii) carrying out a pilot study to identify the domain structure of a measure and establishing the psychometric properties of the instrument (e.g. as part of demonstrating aspects of reliability, validity and responsiveness) , establishing other key aspects such as costs and generating stakeholder feedback [23, 24, 25]; and (iv) piloting the PROM as part of a feasibility study to inform the design of a larger definitive randomised control trial . With the rise in ePROMs, there is also more demand to undertake feasibility testing to test the equivalence of administering an e-version of a PROM compared to its paper version (although the need for such testing has been questioned [32, 33]) and/or establish its acceptability and usability as part of routine clinical care [34, 35].
Despite the essential role that pilot and feasibility testing plays in relation to PROMs, little attention to date has been given to this important stage. It would seem that there are missed opportunities to provide more guidance and ideas regarding what types of pilot or feasibility tests could be carried out. In particular, this is in relation to the ways in which pilot and feasibility tests can be integrated during PROM development, evaluation and implementation and also in terms of what the “standards” are for assessing the methodological quality of these types of studies.
This series seeks to provide a forum for the research community to share and disseminate the work they have been doing which concerns the pilot and feasibility testing of PROMs. This is often reported as one small step during the PROM reporting process and may be the result of peer review journals not considering this stage of high enough importance to dedicate publication space. Therefore, this series aims to give researchers the ability to dedicate the reporting space needed to fully report the processes undertaken, raise the profile of pilot and feasibility testing in relation to PROM research and to provide a platform on which innovative methods can be shared. By dedicating more publication space to the reporting of pilot and feasibility studies in relation to PROMs, it may also help to open up a dialogue amongst the PROM research community about some of the academic and practical issues raised above.
In the future, hopefully, the pilot and feasibility testing of PROMs can align more fully with the MRC guidance so that (i) this crucial stage of the research process can be integrated across all aspects of preparatory work related to PROM development, evaluation and implementation and (ii) clearer guidance and benchmarks for conducting such pilot and feasibility studies can become available to support the research community whilst carrying out these important stages during their PROM-based research.
The author would like to acknowledge the support of Professor Gillian Lancaster and Ella Flemyng in the preparation of this editorial.
The author was commissioned to write this article by BMC Pilot and Feasibility Studies and would like to thank Ella Flemyng and Gillian Lancaster for their support. No funding was received.
Availability of data and materials
GJ wrote this Editorial.
The views expressed are those of the author alone.
The author declares that she has no competing interests.
Consent for publication
Ethics approval and consent to participate
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- 1.Albrecht GL. Subjective health assessment. In: Jenkinson C, editor. Measuring health and medical outcomes. London: UCL Press; 1994. p. 7–26.Google Scholar
- 2.World Health Organisation. The first ten years of the world health. London: World Health Organisation; 1958.Google Scholar
- 3.Jenkinson C. Assessment and evaluation of health and medical care: an introduction and overview. In: Jenkinson C, editor. Assessment and evaluation of health and medical care: a methods text. Buckingham: Open University Press; 1997. p. 1–5.Google Scholar
- 4.Wigton RS. Medical applications. In: Brehmer B, Joyce CRB, editors. s. Amsterdam: Elsevier; 1988. p. 227–45.Google Scholar
- 6.Fitzpatrick R, Davey C, Buxton M, Jones DR. Patient-assessed outcome measures. In: Black N, Brazier J, Fitzpatrick R, Reeves B, editors. Health services research methods. London: BMJ Publications; 1998. p. 13–22.Google Scholar
- 10.Goodhart IM, Andrzejowski JC, Jones GL, Berthoud M, Dennis A, Mills GH, Radley SC. Patient-completed, preoperative web-based anaesthetic assessment questionnaire (electronic Personal Assessment Questionnaire PreOperative): development and validation. Eur J Anaesthesiol. 2017;34(4):221–8. Xx.CrossRefPubMedGoogle Scholar
- 12.van de Poll-Franse LV, Horevoorts N, van Eenbergen M, Denollet J, Roukema JA, Aaronson NK, Vingerhoets A, Coebergh JW, de Vries J, Essink-Bot ML, Mols F, Profiles Registry Group. The Patient Reported Outcomes Following Initial treatment and Long term Evaluation of Survivorship registry: scope, rationale and design of an infrastructure for the study of physical and psychosocial outcomes in cancer survivorship cohorts. Eur J Cancer. 2011;47(14):2188–94.CrossRefPubMedGoogle Scholar
- 13.Lane A, Metcalfe C, Young GJ, Peters TJ, Blazeby J, Avery KN, Dedman D, Down L, Mason MD, Neal DE, Hamdy FC, Donovan JL, ProtecT Study group. Patient-reported outcomes in the ProtecT randomized trial of clinically localized prostate cancer treatments: study design, and baseline urinary, bowel and sexual function and quality of life. BJU Int. 2016;118(6):869–79.CrossRefPubMedPubMedCentralGoogle Scholar
- 14.Costal Tirado A, McDermott AM, Thomas C, Ferrick D, Harris J, Edwards A, McAllister M. Using patient-reported outcome measures for quality improvement in clinical genetics: an exploratory study. J Genet Couns. 2017. doi: 10.1007/s10897-017-0079-6 [Epub ahead of print].PubMedPubMedCentralGoogle Scholar
- 15.Mamiya H, Lix LM, Gardner W, Bartlett SJ, Ahmed S, Buckeridge DL. Montreal Accord on patient-reported outcomes use series—paper 5: patient reported outcomes can be linked to epidemiologic measures to monitor populations and inform public health decisions. J Clin Epidemiol. 2017. doi: 10.1016/j.jclinepi.2017.04.018.
- 17.Ahmed S, Ware P, Gardner W, Witter J, Bingham CO 3rd, Khairy D, Bartlett SJ. Montreal Accord on Patient-Reported Outcomes Use Series—Paper 8: patient reported outcomes (PRO) in electronic health records can inform clinical and policy decisions. J Clin Epidemiol. 2017. doi: 10.1016/j.jclinepi.2017.04.011.
- 19.Devlin N, Appleby J. Getting the most out of PROMs: putting health outcomes at the heart of NHS decision making. London: The King’s Fund; 2010.Google Scholar
- 21.Browne J, Jamieson L, Lawsey J, van der Meulen J, Black N, Cairns J, Lamping D, Smith S, Copley L, Horrockes J. Patient reported outcome measures (PROMs) in elective surgery. 2007. Report to the Department of Health; 2007.Google Scholar
- 25.Peters M, Crocker H, Dummett S, Jenkinson C, Doll H, Gibbons E, Luengo-Fernandez R, Fitzpatrick R. Pilot study of patient reported outcome measures (PROMs) in primary care. Oxford: 2013. https://www.qoru.ac.uk/about/research/application/pilot-study-of-patient%E2%80%90reported-outcomemeasures-proms-in-long-term-conditions-in-primary-care/.
- 26.US Department of Health and Human Services Guidance for Industry. Patient-reported outcome measures. Use in medical product development to support labeling claims. Washington, DC: US Food and Drug Administration; 2009.Google Scholar
- 27.Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM, de Vet HC. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539–49.CrossRefPubMedPubMedCentralGoogle Scholar
- 28.Mokkink LB, Terwee CB, Gibbons E, Stratford PW, Alonso J, Patrick DL, Knol DL, Bouter LM, de Vet HC. Inter-rater agreement and reliability of the COSMIN(COnsensus-based Standards for the selection of health status Measurement Instruments) checklist. BMC Med Res Methodol. 2010;10:82.CrossRefPubMedPubMedCentralGoogle Scholar
- 30.Poku E, Duncan R, Keetharuth A, Essat M, Phillips P, Woods HB, Palfreyman S, Jones G, Kaltenthaler E, Michaels J. Patient-reported outcome measures in patients with peripheral arterial disease: a systematic review of psychometric properties. Health Qual Life Outcomes. 2016;14(1):161. Review.CrossRefPubMedPubMedCentralGoogle Scholar
- 31.Vernon-Wilson L. The Warwick-Edinburgh Mental Well-being Scale (WEMWBS) a pilot study of utility and feasibility in older aged people’s mental health services. New Forest East Community Mental Health Treatment Team: 2013.Google Scholar
- 32.Muehlhausen W, Doll H, Quadri N, Fordham B, O’Donohoe P, Dogar N, Wild DJ. Equivalence of electronic and paper administration of patient-reported outcome measures: a systematic review and meta-analysis of studies conducted between 2007 and 2013. Health Qual Life Outcomes. 2015;13:167.CrossRefPubMedPubMedCentralGoogle Scholar
- 33.Bjorner JB, Rose M, Gandek B, Stone AA, Junghaenel DU, Ware Jr JE. Difference in method of administration did not significantly impact item response: an IRT-based analysis from the Patient-Reported Outcomes Measurement Information System (PROMIS) initiative. Qual Life Res. 2014;23(1):217–27.CrossRefPubMedGoogle Scholar
- 34.Ashley, L., Jones, H., Forman, D., Newsham, A., Brown, J., Downing, A., … Wright, P. Feasibility test of a UK-scalable electronic system for regular collection of patient-reported outcome measures and linkage with clinical cancer registry data: the electronic Patient-reported Outcomes from Cancer Survivors (ePOCS) system. BMC Med Inform Decis Mak. 2011;11:66Google Scholar
- 35.Andikyan V, Rezk Y, Einstein MH, Gualtiere G, Leitao Jr MM, Sonoda Y, Abu-Rustum NR, Barakat RR, Basch EM, Chi DS. A prospective study of the feasibility and acceptability of a web-based, electronic patient-reported outcome system in assessing patient recovery after major gynecologic cancer surgery. Gynecol Oncol. 2012;127(2):273–277.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.