Publications reporting discrete choice experiments of healthcare interventions rarely discuss whether patient and public involvement (PPI) activities have been conducted. This paper presents examples from the existing literature and a detailed case study from the National Institute for Health Research-funded PATHWAY programme that comprehensively included PPI activities at multiple stages of preference research. Reflecting on these examples, as well as the wider PPI literature, we describe the different stages at which it is possible to effectively incorporate PPI across preference research, including the design, recruitment and dissemination of projects. Benefits of PPI activities include gaining practical insights from a wider perspective, which can positively impact experiment design as well as survey materials. Further benefits included advice around recruitment and reaching a greater audience with dissemination activities, amongst others. There are challenges associated with PPI activities; examples include time, cost and outlining expectations. Overall, although we acknowledge practical difficulties associated with PPI, this work highlights that it is possible for preference researchers to implement PPI across preference research. Further research systematically comparing methods related to PPI in preference research and their associated impact on the methods and results of studies would strengthen the literature.
We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
Despite growing recognition of the potential benefits of patient and public involvement (PPI) and the formal requirement by many funders to include PPI in research, we found a limited number of preference studies that utilised PPI activities.
The case study demonstrates that PPI activities can be beneficial across the stages of preference research.
Using the case study and published literature, we highlight that there are many potential stages for PPI activities in preference research that will achieve a potentially impactful and less tokenistic level of engagement.
Production of research and its subsequent translation into practice can be a challenging process. A key facilitator to ensuring wider relevance of research is the involvement of stakeholders from the start of the research, and this approach is increasingly promoted across different areas of healthcare in the production of healthcare  and by funders of research . In healthcare, the involvement of public stakeholders is commonly referred to as ‘Patient and Public Involvement and/or Engagement’ (PPI or PPIE) . The encouragement of PPI is often based on the principle that patients and the public are key consumers of healthcare and therefore benefit from healthcare research [3, 4]. In creating a partnership between researchers and the public or patients, it is anticipated that research becomes more relevant to the potential beneficiaries . Throughout the paper, when we discuss PPI we refer to the INVOLVE definition as “public involvement in research as research being carried out ‘with’ or ‘by’ members of the public rather than ‘to’, ‘about’ or ‘for’ them” .
Health economics is a broad discipline concerned with efficiency, effectiveness, behaviour and equity. As such, there are many ‘non-research’ stakeholders to consider; for example, patients, caregivers, clinicians and policy makers in local and national government institutes or agencies all of whom may have an interest in the development or results of research questions. A key fundamental of health economics is to maximise benefits (often health) to a certain population usually under constrained resources . Consequently, much research conducted by health economists may have an impact in the ‘real world’. For example, research may affect how hospitals are organised or inform decisions by the National Institute for Health and Care Excellence that may affect patients’ access to medicines [8, 9]. To maximise benefits in markets that are imperfect, economists may use stated preference methods to quantify the value placed on goods or services . Discrete choice experiments (DCEs) have been applied in health since the 1990s . In a DCE, individuals select their preferred option from a series of hypothetical goods or services, which are described in terms of fixed attributes but vary in their levels . The choices made reveal the trade-offs they are willing to make and therefore the relative importance of the attributes in their decision making. Recent reviews have shown DCEs to be used in a wide range of applications from labour market choices to health-state valuations [11, 13].
The results of DCE research are having an increasing impact on healthcare practice and decision making. At the ‘micro-level’, preference data are informing shared decision-making tools to help patients and clinicians make informed choices [14,15,16]. At the ‘macro-level’, regulatory decisions regarding product safety are being made on behalf of whole populations based on benefit-risk evidence produced by DCE studies [17,18,19]. For example, the US Food and Drug Administration is developing guidance for quantitative preference elicitation studies  and the European Medicines Agency is involved with the Innovative Medicines Initiative PREFER project, which aims to increase the role of quantitative patient preference data throughout the drug life cycle . Results from preference studies can also be submitted alongside other types of evidence as part of a health technology assessment . Understanding preferences through the use of DCE research can also help to align healthcare interventions to preferences .
There are now many best practice guidelines for conducting DCEs [23, 24] including different aspects of design  and analysis . There are also guidelines specifically for conducting qualitative research to develop preference instruments . However, there are no specific guidelines about how to conduct effective PPI throughout the preference study process. Note that typically, patients informing the design of experiments fall under the heading of qualitative research, for example, as part of focus groups  or interviews . Patient and public involvement specifically refers to members of the public carrying out research with or without the support of other researchers, which is a two-way process (if researchers are involved), compared to qualitative research in which the researchers are informed by participants . The literature notes that there remains uncertainty in how to incorporate PPI into healthcare research, which may lead to suboptimal impact and a feeling that engagement was purely tokenistic .
We first present examples from the existing published literature, describing the key stages of involvement outlined by the authors and any benefits or challenges discussed. We then consider in more detail a DCE case study taken from a recent National Institute for Health Research-funded research programme aimed at improving psychological outcomes in cardiac rehabilitation (CR), which demonstrates that PPI can be used at several stages throughout the design, creation, recruitment and reporting of preference research. In addition, we detail the benefits and challenges of including PPI in the case study research. Finally, drawing together the existing literature, the case study and wider PPI literature, we describe how PPI activities may provide insights at each stage of the preference study process to encourage researchers to consider PPI at all stages of the research, moving away from minimal tokenistic engagement.
2 Examples from Stated Preference Literature
Examples of PPI activity informing preference research were identified from the published literature by searching for keywords, such as “patient involvement”, “public involvement”, “stakeholder involvement” and “stakeholder engagement”, in studies identified by a previous systematic review of DCEs (searches updated to capture evidence published up to January 2018) . Note that full details on the methods and results for the review can be identified in the referenced published paper. Over 300 health-related DCEs have been published since 2013, however, we only identified a handful of examples that described the stakeholder engagement (including patient representatives and groups) in the article or report. An overview of these studies, with a focus on engagement activities, is provided in Table 1. The identified studies did not necessarily refer to PPI activities, rather patients and the public (including caregivers) were among key groups included in wider stakeholder engagement activities and alternative terminology was often used.
The identified examples did not typically provide a thorough overview of the benefits and challenges of engagement; however, this is understandable given word limits on journal articles and that the main focus of the authors would be communicating key methods and study results. The extent of engagement across stages of preference research varied, most commonly it was used to develop attributes, although there are examples of it being used at other stages (e.g. improving the readability of survey materials and interpreting results).
The papers noted some limitations of the stakeholder engagement; including challenges interpreting attribute terminology, missing attributes, the time taken for the process, not all members attended every meeting and potential bias in the sample/representativeness of the sample [34, 35, 37]. Though it was noted in the papers, missing attributes is not necessarily specific to stakeholder engagement, as it is possible that this may occur if stakeholder engagement is not used to assist in identifying attributes and attribute development also needs to consider feasibility (limits on the number of attributes). Furthermore, using stakeholder engagement alongside other methods (e.g. literature review and qualitative research) may reduce the likelihood of missing attributes by considering a wider perspective. All identified studies were positive about stakeholder engagement, despite limitations. Peay discussed that although the design of such an experiment is complex, it can be led by an advocacy organisation (with expert collaboration) and noted that the community engagement was particularly advantageous in achieving recruitment targets . dosReis et al. commented that stakeholder advisors were well informed and knowledgeable, and subsequently their inclusion as coinvestigators added a depth of understanding to enhance the research . Wittenberg  provides a useful commentary on the benefits of stakeholder engagement in stated preference research, focusing on the study by Janssen et al. .
3 Case Study
The literature review presents some useful examples demonstrating potential stages of involvement throughout the process of preference research and an overview of potential limitations and benefits. However, the literature presented limited detail on involvement activities. The case study below reports PPI across the stages of a DCE aiming to provide a more thorough overview of the aspects of involvement and the associated benefits and limitations.
3.1 Study Objectives
The PATHWAY Programme is a 6.5-year project funded under the UK National Institute for Health Research Programme Grants for Applied Research (RP-PG-1211-20011) . The study aims to improve access to more effective psychological interventions for patients attending CR who present with symptoms of depression and/or anxiety. As part of the PATHWAY programme of work, a multicentre, two-arm, single-blind, randomised controlled trial is being conducted to compare the clinical and cost effectiveness of group-based metacognitive therapy plus CR with CR alone. In addition to the trial, a DCE is being conducted to estimate preferences for psychological care within the CR pathway. This case study focuses on the PPI activities related to the DCE work specifically. The DCE research aims to determine the relative importance of characteristics of the psychological intervention in CR, and to assess how people trade between these characteristics. This information can help services target improvements in CR to the aspects that are most important to current and future potential participants of CR.
3.2 Establishing a Working Partnership
The PPI group (the PATHWAY Patient and Public Advisory Group) was formed to provide support, guidance and advice at all stages of the research (not just the DCE activities), and to ensure that the study aims, procedures and dissemination were informed by those with lived experience. To be eligible as a member of the PPI group, potential members had to have experience of one of the following: heart disease, anxiety and/or depression, or being a carer of someone with one or more of these conditions. The group were originally recruited via patient networks, such as Salford Citizen Scientist, the Ticker Club and Salford Heart Care (advertisement included in the Electronic Supplementary Material [ESM]). Initially, there were ten PPI members, which included a chair member who was involved in the grant application; however, two members have left, and another has stepped back, the reason typically given for withdrawing from the PPI group was ill health. Of the seven current members of the PPI team, the majority are female and over the age of 65 years (further details included in the ESM). During the PPI activities related to the DCE, the majority were male, which is aligned with the general population undergoing CR . The PPI group is coordinated and supported by the PPI lead, with meetings held two or three times per year. Patient and public involvement across the whole project will be reported in a separate paper.
3.3 Stages of Patient and Public Involvement Activity
An overview of the PPI activity in the preference work is presented in Fig. 1 and took place over five sessions conducted with the PPI group. The stages of PPI activity were determined through researcher experience and discussion with PPI coordinators. The research team was keen to include PPI throughout the process.
As a first step, the group was given introductory training focused on health economics and stated preference survey techniques. Members were introduced to the reasons for conducting and the benefits of preference research. To illustrate choice sets, two examples from the literature were used; one looking at preferences for health states and a second focused on preferences for primary care consultations [42, 43]. Following training, the group was asked to focus on the research question, which they agreed was clear. However, the PPI group recommended that the term metacognitive therapy was replaced with psychological therapy, as it would be a more familiar term and therefore prevent confusion. This was agreed by the research team as results are likely to be relevant to psychological therapies more widely.
The biggest area of PPI activity was step 3 (developing the choice questions); the first stage of this work involved splitting into two smaller groups to generate initial ideas on what members might consider or want to know prior to attending therapy (e.g. location/setting, information provided prior to starting therapy and time required), with refinement in later sessions to reduce the number of attributes and levels down. Prompts for potential attributes were provided based on expert opinion, the design of psychological therapy, qualitative interview feedback and a review of the literature. In the case study, a separate qualitative component of work was conducted; however, the primary aim was not to inform the DCE design, rather to explore the impact of cardiac events on patients and their psychological needs [44, 45]. However, qualitative interview feedback was reviewed and supported the DCE design by providing some initial ideas for attributes. When discussing key attributes, there were some instances where the PPI group disagreed on the importance of attributes, and a majority consensus was reached. Note that the PPI group found it clearer if the researchers referred to “characteristics” rather than “attributes” and therefore this language was used in the PPI activities and survey development. A summary of step 3 and step 4 (demographics) is provided in the ESM.
Once survey materials were drafted, following review and revisions by clinical and academic experts, PPI members completed them independently to review and provide feedback on contents and clarity. This also helped to inform the estimate for the time taken to complete the survey. One useful idea from the PPI group was that reading from white paper can be challenging for people with dyslexia and offering coloured paper copies of surveys may be beneficial. The majority of participants in the planned survey are likely to be identified via the PATHWAY trial and a market research company; however, in the case of issues with recruitment (e.g. slow recruitment or a sample not representative of the population), the PPI members were asked to provide a list of recommended patient groups to target to ensure recruitment targets are met.
Finally, as part of the wider PPI work, the group was asked to co-produce the dissemination plan for the PATHWAY trial. The group identified key messages, target audiences, formats, publications/venues and suggested approaches to develop a series of ‘off-the-shelf’ dissemination ‘products’ for a public audience. These suggestions informed draft products that the group then modified through group discussion and e-mail. This process can be started in advance of study results allowing time for discussion around the selection and translation of results before the end of the project. It is worth noting that while the group ultimately decided to focus on the development of public facing dissemination, involving them in the initial idea generation for all target audiences can be helpful in generating novel ideas.
3.4 Challenges and Limitations
The debate between PPI members during group activities demonstrated the complexities of choices and showed that participant characteristics will likely impact choices in a real-world setting. Managing disagreements in a PPI group can be challenging when members have different views and when some members of the group are more vocal than others. However, by splitting the group into two (approximately four members in each), it was kept manageable and everyone had an opportunity to feedback their opinion. In addition, a further session was needed to refine the list of characteristics of therapy (attributes for inclusion in the DCE design) as initially it was an unmanageable size. Note, these challenges may be more prominent when the groups are diverse or larger. It is recommended that preference researchers ensure a PPI co-ordinator is present at meetings as they provide vital assistance when dealing with any challenges and facilitate communication between researchers and PPI members. A further challenge was the length of time between meetings and unavoidable absences, especially from the first meeting. This resulted in the need to recap the training on preference research. While an effective solution, this did have an impact on the time available for activities within the meeting. As with all engagement work, the views of PPI participants may not be representative of the wider population. Though the group was typical of the population undergoing CR (typically retired and a higher proportion of male individuals) during DCE-related activities, it is important to consider that feedback will not reflect the total CR population . Additionally, because members have withdrawn owing to ill health, the structure of the group has changed (the majority are now female).
3.5 Concluding Thoughts
It was anticipated that the PPI group might find the training and activities around the preference research uninteresting; however, this was not the case. When given the opportunity to feedback on their involvement in the wider PATHWAY project, the PPI members remarked that they felt that one of their areas of greatest impact had been related to the preference work. In contrast, with using a purely qualitative approach to identifying attributes and levels for a DCE, the inclusion of PPI (seeing the PPI as research collaborators, rather than research subjects as they would be in qualitative research) had benefits. In particular, it was an iterative process that allowed researchers to obtain feedback at several stages of design and the PPI group had an introduction to preference research and felt suitably informed and involved. Though it may be seen as less systematic than a qualitative research study with a more rigorous protocol, there were benefits to the less formal approach, such as having the ability to go back to the PPI team to reduce the number of attributes and levels. Overall, activities with the group were successful; PPI members drew on their experience to identify a range of characteristics that people attending CR may consider before deciding on whether to attend metacognitive therapy sessions and to identify the participant characteristics likely to have an impact on choices. It was noted by the wider PATHWAY team not only to be useful for the DCE design, but for wider practice and policy, especially when considering the low uptake rates associated with CR .
4 Fostering Patient and Public Involvement in Stated Preference Research
A stated preference study has multiple inter-related steps from developing the choice tasks to constructing the experimental design and collecting data from respondents for statistical analyses. In this section, we describe how PPI activities may provide insights at each stage of the preference study process, informed by the examples from the literature, the case study and discussion between authors (preference researchers and a PPI coordinator). This is not to say that PPI must be included across all stages, rather to provide researchers with an overview of how PPI can be useful across the process of preference research and to encourage researchers to move away from including PPI at a single stage, which can seem tokenistic. Furthermore, we recognise that the usefulness of PPI will vary according to the research question, methods (e.g. experiment design) and existing literature.
The case study highlighted the importance of introductory training about health economics and preference research to the PPI group, as well as recap sessions if the design process is lengthy. It is recommended that future studies consider optional recap sessions prior to activities. An example of how health economics can be communicated to PPI researchers is included in the book “A Research Handbook” .
4.1 Defining the Research Question
With any research project, the first stage is often developing and defining the research question . In some instances, this may be defined by a particular policy but in many instances specifying the study perspective and rationale for research may be difficult, particularly if the results of the research have relevance for multiple groups, e.g. patients and their caregivers. A review of PPI in health and social care research identified that it can help to contribute towards user-focused research objectives and user-relevant research questions . In addition, it is important for the research question to be clear and understandable to potential participants, PPI can help to ensure participants clearly understand the aim of the research they are contributing to, as demonstrated in the PATHWAY case study.
4.2 Selecting Attributes and Levels
There is some consensus that attributes and levels should be identified using qualitative research methods with individuals that represent the sample for the final stated preference survey . Using PPI activities to identify attributes and levels may have parallels to using qualitative research methods in this context. For example, researchers collaborating with stakeholders should take a rigorous approach, documenting or recording changes to the research pathway and why these occurred, as well as reflecting on their own biases and how these may influence participants’ contributions. However, there may be some advantages to using stakeholder engagement in addition to purely qualitative approaches [34, 35, 37, 38]. For example, advisory boards may provide a useful opportunity to elicit other practical insights regarding implementation that may not emerge in qualitative research. dosReis et al.  engaged stakeholders in the analysis of qualitative data collected to identify attributes and levels. Engaging stakeholders in this capacity may provide an opportunity to pragmatically reduce many themes emerging in expansionary qualitative research to a reasonable and plausible (not just desirable) set for use in a choice experiment. Seeing PPI as a two-way communication could be a more iterative process that allows for greater discussion and debate around design (vs a one-way qualitative method of obtaining public feedback). Note that we are not suggesting that PPI replaces qualitative methods, rather that it can add to several stages of preference research.
4.3 Choice Set Design
Stakeholder engagement may assist at the early stages of the study to understand the most appropriate choice format (e.g. contingent valuation, DCE, best–worst scaling). For example, Coxon et al. chose a ‘partial profile’ design for their DCE study after the research user group expressed that six attributes were too cognitively burdensome . Another key consideration for many researchers is whether to have a forced-choice experiment. To relax the assumption positive demand (i.e. a respondent will always choose to consume one of the alternatives presented), choice experiments may include an ‘opt-out’ or ‘status quo’ option. However, there is evidence to suggest that individuals can bring different meaning to the ‘none’ alternative . Stakeholder engagement exercises may be useful in revealing the specification (levels, terminology) for an opt-out alternative.
4.4 Design of Survey Materials
Qualitative interviews to formally pre-test the survey instrument are likely to be seen by many health preference researchers as the ‘gold standard’ of design. However, an initial survey needs to be developed for use in any pilot work. Patient and public involvement activities may be an ideal opportunity to refine language and terminology to encourage respondents to correctly and consistently interpret descriptions and questions. Seo et al.  used two committees comprising experts and community stakeholders to ensure descriptions were appropriate to patient respondents completing the survey but would also resonate with the medical community. In the PATHWAY case study, members of the PPI group were given copies of the draft stated preference survey materials and made some interesting recommendations. Some were presentation related, such as printing on coloured paper to help people with visual impairments and bolding text with the closing date, others focused on simplifying the wording and some were more practical, e.g. telling people when to expect reimbursement. This also allowed the team to receive initial feedback on how long the survey may take. A review of PPI in randomised controlled trials described that one of the most reported benefits was related to writing participant information sheets, such as improving the clarity of information , another review explained that it can help to develop user-friendly information, questionnaires and interview schedules .
4.5 Data Collection
A key challenge in preference research is recruiting an adequate sample . Stakeholder engagement groups may be able to provide practical advice and personal insights into different recruitment strategies. For example, community leaders have been used to create and distribute newsletter notices and recruitment e-mails and raise awareness through word-of-mouth recruiting in health preference research . Patient and public involvement activities have been linked to improved recruitment materials , more appropriate recruitment strategies for studies  and an increased likelihood of recruiting the necessary sample within restricted timeframes [50,51,52,53]. In addition, stakeholder groups may be able to advise on approaches to recruit lesser reached groups or provide advice on appropriate introductions or reimbursement. In a previous stated preference study by one of the authors, the involvement of a service user researcher was imperative to obtaining a sufficient sample size of secondary care users of mental health services, to assess individual preferences for key aspects of care planning . This included the service user researchers making use of their network, contacts and known organisations to boost recruitment. In addition, in the PATHWAY case study, PPI members were keen to ensure that the sample reached would include older people who do not regularly use social media or who may not see posters in clinics and therefore recommended attending patient groups in the community, such as The Ticker Club. It must be noted that PPI advice around design and recruitment must be considered alongside the timeframe and budget for the study.
4.6 Interpreting the Results
Involving the PPI group in the results stage of preference research can have several benefits. The PPI members can check that the results are clear and understandable to a lay audience. In addition, the group can help to interpret results, especially anything that seems unexpected. Adams et al. explored a surprising finding that some parents preferred having no financial incentives for vaccinating children. The parent advisory group suggested that this could be seen as a reward for ‘bad behaviour’ for those who had not vaccinated their child prior to the introduction of a financial incentive . In the previously discussed example with secondary care users of mental health services (assessing individual preferences for key aspects of care planning), there was one unusual result,in which participants preferred a level “agree” to “completely agree” for risk management . It had been assumed that there would be a linear trend for preferences (i.e. completely agree would be favoured over agree). However, the service user researcher was able to help interpret as they were aware that participants may be wary of the negative connotations of risk and placing too much emphasis on this aspect,therefore, whilst service users would still want to see risk management considered, they would not want it to be the focus. This is something that the research team would have been unlikely to explain without the assistance of a service user. The PATHWAY case study has not yet reached this stage, however, during the design of attributes and levels, the PPI feedback was recorded and will be used to support the interpretation of results.
4.7 Generating Impact
There are a number of ways that PPI members can be involved in the dissemination phase; from summarising the findings in an accessible way to supporting as co-authors on an academic paper. In addition, they can provide ideas that help the research to reach a wider and more diverse audience (such as patient groups or forums). Morgan et al. describe how the results of a poster presentation of their results was shown to the mother-and-baby group . The group offered suggestions about how the research could be disseminated including key websites, leaflets and newsletters. The literature reviewed in this article also recognises that the involvement of PPI groups in dissemination may result in more innovative dissemination activities, which can help to increase impact, such as social media blogs . The PATHWAY PPI members have generated ideas for dissemination. The literature notes that PPI in dissemination is most likely to impact on implementation and changes to practice .
Patient and public involvement and related activities provide opportunities to collaborate with key stakeholders to guide the development of research and ensure the outputs are relevant to those who may be affected. Acting on opportunities to engage patients and/or the public in a collaborative sense rather than using these individuals as research subjects may yield different but equally interesting insights. This paper outlines potential stages of PPI activity across the design, recruitment and dissemination processes related to preference research to support researchers and to encourage PPI in this work. Note that the paper focuses on DCE research as the case study used is a DCE, however, we expect that the findings are relevant and applicable to researchers involved in a broad range of preference research, particularly those using hypothetical survey-based approaches.
There is the need to avoid a sense of tokenistic engagement, which has been associated with funders requiring PPI [5, 52]. This may be in part owing to a feeling of uncertainty around how and when to involve PPI . We would encourage all researchers to ensure that PPI activity will be used to inform the research before patient and public contributors are contacted. Based on the published literature, it appears PPI has been rarely reported in DCE studies despite PPI being a recommended or required at all stages of research by funders . This suggests that PPI is not always included as part of preference research, or that authors feel it did not make a substantial enough contribution to the research to warrant reporting (i.e. it was perceived as a “tick box” exercise to appease funders). Additionally, the identified examples use engagement activities at a limited number of stages across the research. We outline several stages in which PPI can contribute to preference research, which may help researchers to create a more comprehensive and less tokenistic sense of engagement, where the PPI group are clearly contributors/collaborators to research.
The existing literature discusses the many possible benefits from PPI activities, including clearer survey materials, timely recruitment and innovative dissemination. In addition, we feel that specific to preference research, PPI members may be able to offer practical insights and pragmatic approaches to refining themes for inclusion in preference studies and furthermore, can make valuable observations that support researchers interpreting results. We anticipate that the usefulness and opportunities for PPI activities will vary depending on the precise study research questions and study design. However, researchers will not always be aware of the potential benefits of PPI in the planning stages. For example, in Sect. 4.6, there are two examples presented where it helped to interpret unexpected results from preference research [54, 58]. Although we encourage researchers to think carefully about the incorporation of PPI and to ensure that PPI activities inform the research (i.e. the PPI groups time is not wasted on meaningless activities), the full benefits will not always be known until the end of the research. Researchers must consider the feasibility (e.g. time and cost restrictions) and the potential benefits of PPI when planning a study.
There are challenges associated with PPI activity, typically focused on time and funding requirements [5, 52]. Wider recent reviews have noted inconsistencies in the reporting of PPI activities and that researchers report difficulties in providing information on PPI activities, which perhaps can partially explain the lack of examples identified [5, 59]. New checklists, such as the GRIPP2, can support and encourage researchers to more comprehensively report PPI activities in preference research . Inadequate management of expectations has been noted as a potential barrier to successful PPI and we recommend that researchers and PPI members outline expectations prior to conducting the work, to ensure a shared vision of the research . Recommendations for managing expectations are outlined in the wider PPI literature . Further challenges include difficulties translating complex terminology and teaching groups adequately to allow them to contribute more thoroughly , low meeting attendance or PPI members dropping out (illustrated in the case study) . Finally, caution needs to be taken as groups may not always be representable of the wider population and members may act in their own interests or provide biased feedback .
There are some limitations to this work, most notably, recommendations based on author opinion/reflection have been drawn using a single case study and examples from the published literature. As PPI is encouraged by organisations and funders, it is likely that there will be a growth in the relevant literature in the future that may strengthen processes. Therefore, we recommend researchers keep up to date and ensure they learn from new examples to ensure they use PPI activities most effectively. To identify examples from the literature, the authors used results from a previous systematic review and searched the included publications for key terms related to PPI . The search was updated to identify more recent results, but this was not systematic, and the terms used to identify examples (e.g. “patient involvement”, “stakeholder engagement”) were kept broad to account for differences in terminology but may not have captured every term used to describe PPI. Therefore, it is possible that the review did not identify every example from the literature. However, as the purpose was to identify some existing published examples and to summarise these, this is unlikely to affect the conclusions of this paper. A full systematic review of PPI contributions to preference research would incur several challenges around the differences in terminology. Additionally, it is outside the scope of this paper. This paper focuses on how PPI can inform DCE research. It does not consider how best to carry out PPI activities, for example, choosing PPI members, effective facilitation and dealing with challenges. We recommend that DCE researchers work closely with an experienced PPI facilitator and review the wider PPI literature related to effective practices. In addition, the paper focuses on a UK setting because of the use of the case study conducted in England and author experience. However, we think that it is possible for researchers across all countries to include PPI at multiple stages of preference studies and think the ideas presented will be generalisable to researchers in other countries. Preferred terminology and methods related to PPI activities and preference research more widely are likely to vary according to country and/or funders. Researchers will need to consider what is appropriate in the context of their own study and setting. The case study presented benefitted from having funders who encourage PPI and a PPI coordinator to support work. The feasibility, relevance and potential for benefit related to PPI activities is likely to differ across specific projects and requires careful consideration from researchers. Finally, the role of this paper is to support researchers planning PPI in preference research. As the use of PPI in preference research expands, further research systematically comparing methods related to PPI in preference research and how they impact on the agreed methods and results of studies would strengthen the literature.
As discussed, the examples presented demonstrated that PPI can be very useful in preference research. Initial concerns that the case study PPI members might find the DCE research uninteresting were unfounded. It is also worth remembering that there are benefits for PPI members, for example, a project in cancer research discussed that participants valued the opportunity they had to take part in research and the knowledge and skills they acquired throughout the process . The PATHWAY PPI members have also noted that they benefitted from being part of the group as it provided an opportunity to talk to others who had been through similar experiences, they learnt new information and they felt they “had a voice”. Therefore, researchers should be encouraged, knowing that PPI members will also benefit from robust research processes.
Availability of Data and Materials
Materials used for this study are available from the corresponding author on reasonable request.
Staniszewska S, Denegri S, Matthews R, Minogue V. Reviewing progress in public involvement in NIHR research: developing and implementing a new vision for the future. BMJ Open. 2018;8:e017124. https://doi.org/10.1136/bmjopen-2017-017124.
Caress A, Ford A, Roberts L, Turner K, Ward D, Williamson T. Briefing notes for researchers: public involvement in NHS, public health and social care research. INVOLVE. 2012. Available from: https://www.invo.org.uk/wp-content/uploads/2014/11/9938_INVOLVE_Briefing_Notes_WEB.pdf. Accessed 30 June 2020.
National Institute for Health and Care Excellence. Patient and public involvement policy. Available from: https://www.nice.org.uk/about/nice-communities/nice-and-the-public/public-involvement/public-involvement-programme/patient-public-involvement-policy. Accessed 16 Jan 2020.
Minogue V, Cooke M, Donskoy AL, Vicary P, Wells B. Patient and public involvement in reducing health and care research waste. Res Involv Engagem. 2018;4:5. https://doi.org/10.1186/s40900-018-0087-1.
Blackburn S, McLachlan S, Jowett S, Kinghorn P, Gill P, Higginbottom A, et al. The extent, quality and impact of patient and public involvement in primary care research: a mixed methods study. Res Involv Engagem. 2018;4:16. https://doi.org/10.1186/s40900-018-0100-8.
INVOLVE. What is public involvement in research? Available from: https://www.invo.org.uk/find-out-more/what-is-public-involvement-in-research-2/. Accessed 1 Dec 2019.
Drummond M, Schulpher M, Claxton K, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. 4th ed. Oxford: Oxford University Press; 2015.
Dakin H, Devlin N, Feng Y, Rice N, O’Neill P, Parkin D. The influence of cost-effectiveness and other factors on nice decisions. Health Econ. 2015;24:1256–71. https://doi.org/10.1002/hec.3086.
Kreif N, Grieve R, Hangartner D, Turner AJ, Nikolova S, Sutton M. Examination of the synthetic control method for evaluating health policies with multiple treated units. Health Econ. 2016;25:1514–28. https://doi.org/10.1002/hec.3258.
Mahieu P-A, Andersson H, Beaumais O, Crastes dit Sourd R, Hess S, Wolff F-C. Stated preferences: a unique database composed of 1657 recent published articles in journals related to agriculture, environment, or health. Rev Agric Food Environ Stud. 2017;98:201–20. https://doi.org/10.1007/s41130-017-0053-6.
Soekhai V, de Bekker-Grob EW, Ellis AR, Vass CM. Discrete choice experiments in health economics: past, present and future. Pharmacoeconomics. 2019;37:201–26. https://doi.org/10.1007/s40273-018-0734-2.
Ryan M, Gerard K. Using discrete choice experiments to value health care programmes: current practice and future research reflections. Appl Health Econ Health Policy. 2003;2:55–64.
Clark MD, Determann D, Petrou S, Moro D, de Bekker-Grob EW. Discrete choice experiments in health economics: a review of the literature. Pharmacoeconomics. 2014;32:883–902. https://doi.org/10.1007/s40273-014-0170-x.
Flynn T, Huynh E, Sydney N, Corke C, Health B, Hawkins G. Best-worst scaling allows us to quantify attitudes as well as preferences; response times tells us which are “gut” attitudes with no predictive power. In: Medical decision meeting. Miami; 20th Oct 2014.
Dowie J, Kaltoft MK. Translating the results of discrete choice experiments into p-/e-/m-health decision support tools. Stud Health Technol Inform. 2019;261:193–8. https://doi.org/10.3233/978-1-61499-975-1-193.
Mühlbacher AC, Juhnke C. Patient preferences versus physicians’ judgement: does it make a difference in healthcare decision making? Appl Health Econ Health Policy. 2013;11:163–80. https://doi.org/10.1007/s40258-013-0023-3.
Vass CM, Payne K. Using discrete choice experiments to inform the benefitrisk assessment of medicines: are we ready yet? Pharmacoeconomics. 2017;35:1–21. https://doi.org/10.1007/s40273-017-0518-0.
Medical Device Innovation Consortium (MDIC). Patient centred benefit-risk project report: a framework for incorporating information on patient preferences regarding benefit and risk into regulatory assessments of new medical technology. 2017. Available from: https://mdic.org/project/patient-centered-benefit-risk-pcbr/. Accessed 10 July 2020.
Mott DJ. Incorporating quantitative patient preference data into healthcare decision making processes: is HTA falling behind? Patient. 2018;11:249–52. https://doi.org/10.1007/s40271-018-0305-9.
Ho MP, Gonzalez JM, Lerner HP, Neuland CY, Whang JM, McMurry-Heath M, et al. Incorporating patient-preference evidence into regulatory decision making. Surg Endosc Other Interv Tech. 2015;29:2984–93. https://doi.org/10.1007/s00464-014-4044-2.
Innovative Medicines Initiative. Patient preferences in benefit-risk assessments during the drug life cycle (PREFER) project. Grant Agreement No. 115966. 2016. Available from: https://www.imi-prefer.eu/about/. Accessed 10 July 2020.
Bouvy JC, Cowie L, Lovett R, Morrison D, Livingstone H, Crabb N. Use of patient preference studies in HTA decision making: a NICE perspective. Patient. 2020;13(2):145–9. https://doi.org/10.1007/s40271-019-00408-4.
Bridges JF, Hauber AB, Marshall D, Lloyd A, Prosser L, Regier DA, et al. Conjoint analysis applications in health: a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value Health. 2011;14:403–13. https://doi.org/10.1016/j.jval.2010.11.013.
Lancsar E, Louviere J. Conducting discrete choice experiments to inform healthcare decision making: a user’s guide. Pharmacoeconomics. 2008;26:661–77. https://doi.org/10.2165/00019053-200826080-00004.
Reed Johnson F, Lancsar E, Marshall D, Kilambi V, Mühlbacher A, Regier DA, et al. Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force. Value Health. 2013;16:3–13. https://doi.org/10.1016/j.jval.2012.08.2223.
Hauber AB, González JM, Groothuis-Oudshoorn CGM, Prior T, Marshall DA, Cunningham C, et al. Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force. Value Health. 2016;19:300–15. https://doi.org/10.1016/j.jval.2016.04.004.
Hollin IL, Craig BM, Coast J, Beusterien K, Vass C, DiSantostefano R, et al. Reporting formative qualitative research to support the development of quantitative preference study protocols and corresponding survey instruments: guidelines for authors and reviewers. Patient. 2020;13:121–36. https://doi.org/10.1007/s40271-019-00401-x.
Wittenberg E, Bharel M, Saada A, Santiago E, Bridges JFP, Weinreb L. Measuring the preferences of homeless women for cervical cancer screening interventions: development of a best-worst scaling survey. Patient. 2015;8:455–67. https://doi.org/10.1007/s40271-014-0110-z.
Coast J, Flynn TN, Natarajan L, Sproston K, Lewis J, Louviere JJ, et al. Valuing the ICECAP capability index for older people. Soc Sci Med. 2008;67:874–82. https://doi.org/10.1016/j.socscimed.2008.05.015.
Morgan H, Thomson G, Crossland N, Dykes F, Hoddinott P. Combining PPI with qualitative research to engage ‘harder-to-reach’ populations: service user groups as co-applicants on a platform study for a trial. Res Involv Engagem. 2016;2:7. https://doi.org/10.1186/s40900-016-0023-1.
Ocloo J, Matthews R. From tokenism to empowerment: progressing patient and public involvement in healthcare improvement. BMJ Qual Saf. 2016;25:626–32. https://doi.org/10.1136/bmjqs-2015-004839.
Adams J, Bateman B, Becker F, Cresswell T, Flynn D, McNaughton R, et al. Effectiveness and acceptability of parental financial incentives and quasi-mandatory schemes for increasing uptake of vaccinations in preschool children: systematic review, qualitative study and discrete choice experiment. Health Technol Assess. 2015;19:94. https://doi.org/10.3310/hta19940.
Coxon D, Frisher M, Jinks C, Jordan K, Paskins Z, Peat G. The relative importance of perceived doctor’s attitude on the decision to consult for symptomatic osteoarthritis: a choice-based conjoint analysis study. BMJ Open. 2015;5:10. https://doi.org/10.1136/bmjopen-2015-009625.
dosReis S, Castillo WC, Ross M, Fitz-Randolph M, Vaughn-Lee A, Butler B. Attribute development using continuous stakeholder engagement to prioritize treatment decisions: a framework for patient-centered research. Value Health. 2016;19:758–66. https://doi.org/10.1016/j.jval.2016.02.013.
Janssen EM, Segal JB, Bridges JFP. A framework for instrument development of a choice experiment: an application to type 2 Ddabetes. Patient. 2016;9:465–79. https://doi.org/10.1007/s40271-016-0170-3.
Morgan H, Hoddinott P, Thomson G, Crossland N, Farrar S, Yi D, et al. Benefits of incentives for breastfeeding and smoking cessation in pregnancy (BIBS): a mixed-methods study to inform trial design. Health Technol Assess. 2015;19:1–516. https://doi.org/10.3310/hta19300.
Peay HL. Erratum: a community-engaged approach to quantifying caregiver preferences for the benefits and risks of emerging therapies for Duchenne muscular dystrophy (Clin Ther. 2014;36:624–37). Clin Ther. 2014;36:1313.
Seo J, Smith BD, Estey E, Voyard E, O’Donoghue B, Bridges JFP. Developing an instrument to assess patient preferences for benefits and risks of treating acute myeloid leukemia to promote patient-focused drug development. Curr Med Res Opin. 2018;34:2031–9. https://doi.org/10.1080/03007995.2018.1456414.
Wittenberg E. Instrument development in choice experiments. Commentary on: “applying a framework for instrument development of a choice experiment to measure treatment preferences in type 2 diabetes”. Patient. 2016;9:379–81. https://doi.org/10.1007/s40271-016-0186-8.
Wells A, McNicol K, Reeves D, Salmon P, Davies L, Heagerty A, et al. Improving the effectiveness of psychological interventions for depression and anxiety in the cardiac rehabilitation pathway using group-based metacognitive therapy (PATHWAY Group MCT): study protocol for a randomised controlled trial. Trials. 2018;19:215. https://doi.org/10.1186/s13063-018-2593-8.
British Heart Foundation. National Audit of Cardiac Rehabilitation (NACR) report 2018. Available from: https://www.bhf.org.uk/informationsupport/publications/statistics/national-audit-of-cardiac-rehabilitation-quality-and-outcomes-report-2018. Accessed 17 June 2019.
Norman R, Cronin P, Viney R. A Pilot Discrete Choice experiment to explore preferences for EQ-5D-5L health states. Appl Health Econ Health Policy. 2013;11:287–98. https://doi.org/10.1007/s40258-013-0035-z.
Cheraghi-Sohi S, Hole AR, Mead N, McDonald R, Whalley D, Bower P, et al. What patients want from primary care consultations: a discrete choice experiment to identify patients’ priorities. Ann Fam Med. 2008;6:107–15. https://doi.org/10.1370/afm.816.
McPhillips R, Salmon P, Wells A, Fisher P. Cardiac rehabilitation patients’ accounts of their emotional distress and psychological needs: a qualitative study. J Am Heart Assoc. 2019;8:e011117. https://doi.org/10.1161/JAHA.118.011117.
McPhillips R, Salmon P, Wells A, Fisher P. Qualitative analysis of emotional distress in cardiac patients from the perspectives of cognitive behavioral and metacognitive theories: why might cognitive behavioral therapy have limited benefit, and might metacognitive therapy be more effective? Front Psychol. 2019;9:2288. https://doi.org/10.3389/fpsyg.2018.02288.
Bee P, Brooks H, Callaghan P, Lovell K. A research handbook for patient and public involvement researchers. Manchester: Manchester University Press; 2018.
Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect. 2014;17:637–50. https://doi.org/10.1111/j.1369-7625.2012.00795.x.
Determann D, Gyrd-Hansen D, de Wit GA, de Bekker-Grob EW, Steyerberg EW, Lambooij MS, et al. Designing unforced choice experiments to inform health care decision making: implications of using opt-out, neither, or status quo alternatives in discrete choice experiments. Med Decis Mak. 2019;39(6):681–92. https://doi.org/10.1177/0272989x19862275.
de Bekker-Grob EW, Donkers B, Jonker MF, Stolk EA. Sample size requirements for discrete-choice experiments in healthcare: a practical guide. Patient. 2015;8:373–84. https://doi.org/10.1007/s40271-015-0118-z.
Wilson P, Mathie E, Keenan J, McNeilly E, Goodman C, Howe A, et al. ReseArch with Patient and Public invOlvement: a RealisT evaluation: the RAPPORT study. Southampton (UK): NIHR Journals Library; 2015. https://doi.org/10.3310/hsdr03380.
Bagley HJ, Short H, Harman NL, Hickey HR, Gamble CL, Woolfall K, et al. A patient and public involvement (PPI) toolkit for meaningful and flexible involvement in clinical trials: a work in progress. Res Involv Engagem. 2016;2:15. https://doi.org/10.1186/s40900-016-0029-8.
Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, et al. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014;14:89. https://doi.org/10.1186/1472-6963-14-89.
Ennis L, Wykes T. Impact of patient involvement in mental health research: longitudinal study. Br J Psychiatry. 2013;203:381–6. https://doi.org/10.1192/bjp.bp.112.119818.
Lovell K, Bee P, Bower P, Brooks H, Cahoon P, Callaghan P, et al. Training to enhance user and carer involvement in mental health-care planning: the EQUIP research programme including a cluster RCT. Program Grants Appl Res. 2019;7:1–140. https://doi.org/10.3310/pgfar07090.
McNichol E, Grimshaw P. An innovative toolkit: increasing the role and value of patient and public involvement in the dissemination of research findings. Int Pract Dev J. 2014;4:Article 8.
Staley K. ‘Is it worth doing?’ Measuring the impact of patient and public involvement in research. Res Involv Engagem. 2015;1:6. https://doi.org/10.1186/s40900-015-0008-5.
Andrews LM, Allen H, Sheppard ZA, Baylis G, Wainwright TW. More than just ticking a box…how patient and public involvement improved the research design and funding application for a project to evaluate a cycling intervention for hip osteoarthritis. Res Involv Engagem. 2015;1:13. https://doi.org/10.1186/s40900-015-0013-8.
Adams J, Bateman B, Becker F, Cresswell T, Flynn D, McNaughton R, et al. Effectiveness and acceptability of parental financial incentives and quasi-mandatory schemes for increasing uptake of vaccinations in preschool children: systematic review, qualitative study and discrete choice experiment. Health Technol Assess. 2015;19(94):1–176. https://doi.org/10.3310/hta19940.
Brett J, Staniszewska S, Simera I, Seers K, Mockford C, Goodlad S, et al. Reaching consensus on reporting patient and public involvement (PPI) in research: methods and lessons learned from the development of reporting guidelines. BMJ Open. 2017;7:e016948. https://doi.org/10.1136/bmjopen-2017-016948.
Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ. 2017;358:j3453. https://doi.org/10.1136/bmj.j3453.
Tomlinson J, Medlinskiene K, Cheong VL, Khan S, Fylan B. Patient and public involvement in designing and conducting doctoral research: the whys and the hows. Res Involv Engagem. 2019;5:23.
Pizzo E, Doyle C, Matthews R, Barlow J. Patient and public involvement: how much do we spend and what are the benefits? Health Expect. 2015;18:1918–26. https://doi.org/10.1111/hex.12204.
Thompson J, Bissell P, Cooper CL, Armitage CJ, Barber R. Exploring the impact of patient and public involvement in a cancer research setting. Qual Health Res. 2014;24:46–544. https://doi.org/10.1177/1049732313514482.
We are very grateful to the PATHWAY PPI members (past and present), and the previous PATHWAY PPI co-ordinators Tracey Williamson and Carolyn Gamble for assisting with the design and facilitation of case study activities, as well as the wider PATHWAY team.
This paper presents independent research funded by the National Institute for Health Research under its Programme Grants for Applied Research Programme (Grant Reference Number RP-PG-1211-20011). The views expressed are those of the author(s) and not necessarily those of the National Institute for Health Research or the Department of Health.
Conflict of interest
All authors have completed the International Committee of Medical Journal Editors uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare no support from any organisation for the submitted work (with the exclusion of the stated funder), no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years, and no other relationships or activities that could appear to have influenced the submitted work.
The PATHWAY trial received full ethical approval from the Preston Research Ethics Committee on 6 March, 2015: REC Reference 14/NW/0163. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.
Consent to participate
Consent for publication
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Shields, G.E., Brown, L., Wells, A. et al. Utilising Patient and Public Involvement in Stated Preference Research in Health: Learning from the Existing Literature and a Case Study. Patient 14, 399–412 (2021). https://doi.org/10.1007/s40271-020-00439-2