Skip to main content
Download PDF

Practical Considerations in Using Online Modified-Delphi Approaches to Engage Patients and Other Stakeholders in Clinical Practice Guideline Development

The Patient - Patient-Centered Outcomes Research volume 13, pages 11–21 (2020)Cite this article

Abstract

Patients and caregivers are increasingly recognized as key stakeholders in developing clinical practice guidelines (CPGs). Online engagement approaches offer the promise of a rigorous, scalable, and convenient engagement method. This paper illustrates how an online modified-Delphi approach could be used to engage patients, caregivers, and other stakeholder in CPG development. It provides practical guidance for conducting online modified-Delphi panels that covers (1) joint development of the engagement approach with relevant stakeholders, (2) adaptation of methods used by experts in guideline development, (3) pilot testing, (4) participant recruitment, (5) determining panel size and composition, (6) building participant capacity, (7) facilitation of two-way interaction, (8) continuous engagement and retention of participants, (9) rigorous data analysis, (10) evaluation of engagement activities, and (11) result dissemination. The paper is based on a recently completed study about engaging individuals with Duchenne muscular dystrophy (DMD) and their caregivers in determining the patient-centeredness of DMD care guidelines.

FormalPara Key Points for Decision Makers
Patients, caregivers, and other relevant stakeholders are increasingly engaged in the process of developing clinical practice guidelines.
This paper provides practical guidance on using online modified-Delphi approaches to facilitate engagement of patients, caregivers, and other stakeholders in the guideline development process.
Based on a recent study about engaging individuals with Duchenne muscular dystrophy (DMD) and their caregivers in determining patient-centeredness of DMD care guidelines, we provide 11 practical considerations for using online modified-Delphi approaches for large-scale engagement.

Introduction

Patients and caregivers are increasingly recognized as key stakeholders in developing clinical practice guidelines (CPGs) [1,2,3]. Their involvement could potentially make CPGs more trustworthy, ensure their relevance to patient needs and preferences, facilitate the implementation of guidelines, increase compliance with CPG recommendations, and ultimately improve care quality [4,5,6]. The Institute of Medicine [7], Guidelines International Network [8], National Institute for Health and Care Excellence [9], and other organizations encourage patient and stakeholder involvement in CPG development. Nonetheless, there is little guidance on how best to incorporate patient and caregiver input in CPG development [10]. Methods commonly used to involve stakeholders are including patients and their representatives in guideline working groups, participating in focus groups or individual interviews, and convening a workshop, meeting, or seminar [5]. However, these methods typically require face-to-face interaction and do not allow for large-scale engagement. Guideline groups tend to include one or two patient representatives, and focus groups rarely have more than 11 participants. When patients do participate, they may feel intimidated by clinicians and researchers, especially if the patients are not trained [11].

Online engagement approaches resolve many of these issues. They are scalable and do not require travel to a central location. They are often characterized by the greater openness attributed to anonymous participation [12] among diverse groups of patients and their representatives. Participating from home or other patient-chosen locations makes panels more accessible, particularly if a patient has challenges with travel or public speaking [13]. Online engagement in CPG development may include commenting on draft guidelines, participating in a Delphi process, and using voting tools, Wikis, and discussion forums [1, 3, 14,15,16,17]. An online modified-Delphi approach that combines rounds of rating, anonymous feedback on group results, and a moderated online discussion forum is a promising way to engage large and diverse groups of patients and their representatives [11, 18, 19].

Although this guidance is written primarily for guideline groups, and its objective is to illustrate how to use an online modified-Delphi approach to engage patients and their representatives during different stages of CPG development, we believe it will be useful for large-scale stakeholder engagement in other areas, including prioritizing tasks, creating research standards, and developing healthcare quality indicators [20, 21]. We highlight 11 practical considerations based on our experience in a recent study about engaging individuals with Duchenne muscular dystrophy (DMD) and their caregivers in determining the patient-centeredness of the 2018 DMD care considerations that were published a month before we conducted the online panels [18]. This guidance is also informed by the literature on patient engagement in CPG development [5, 22], our experience conducting more than 25 similarly structured online expert and stakeholder engagement panels, and best practices for conducting Delphi panels [23, 24]. Although we used ExpertLens (an online modified-Delphi system) to collect data, other online platforms with rating, feedback, and discussion functionalities could also be used. Finally, while participants in our study provided input on the patient-centeredness of CPG recommendations, the engagement method could help prioritize guideline topics and intervention outcomes or help determine the extent to which recommendations help ensure health equity.

A Brief Description of an Online Modified-Delphi Approach to Engagement

Direct interaction among participants distinguishes modified-Delphi methods from traditional Delphi panels. Expert panels conducted using the RAND/UCLA Appropriateness Method (RAM) consist of two rating rounds and a face-to-face or phone discussion conducted between the rating rounds [25]. In health services research, RAM is often referred to as a modified-Delphi method because it adds the discussion round. Clinical experts used RAM to develop the 2018 DMD care considerations [26].

Although there are different ways of using the online modified-Delphi method for engaging patients and their representatives in the CPG development process, one way of doing so is to conduct a three- or four-round engagement process to determine the patient-centeredness of draft guideline recommendations using the RAND/PPMD Patient-Centeredness Method (RPM) (Fig. 1) [19].

Fig. 1
figure 1

Source: Khodyakov et al. [19]

The RAND/PPMD patient-centeredness method (RPM)

Full size image

In an optional Round 0 of the RPM, participants are asked about their care preferences, needs, and interests, and the barriers to/facilitators of seeking care. Round 0 is indicated if this information is not available from prior research. Round 0 outcomes can encourage participants in subsequent rounds to think beyond their personal experiences. If needed, participants are also asked to prioritize care outcomes, barriers, and facilitators for a given aspect of care. In Round 1, participants review draft care recommendations, rate them on a predefined set of criteria, such as importance and acceptability (see Box in section 3.1.1), and explain their ratings using open-text boxes. In Round 2, they see how their own Round 1 answers compare with those of the group and whether consensus is achieved. Participants contribute to a moderated, asynchronous, and (partially) anonymous discussion board. Finally, in Round 3, participants can revise their original ratings. The RAM approach to determining group consensus was applied to Round 3 ratings to determine the final group decisions [25].

Practical Considerations for Conducting Online Modified-Delphi Panels

The following practical guidance for using the online modified-Delphi approach covers three stages of stakeholder engagement—preparation, implementation, and evaluation and dissemination—and includes examples from our recent study (Fig. 2).

Fig. 2
figure 2

11 Practical considerations for online modified-delphi panels

Full size image

Preparing for Research

Co-Develop an Engagement Approach with Relevant Patient Representatives

Guideline developers should determine who should be engaged in the CPG process and work with patients, caregivers, and their representatives to design all engagement activities and data collection protocols. At this stage, developers should also consider whether patients may have substantively different perspectives than caregivers and, therefore, whether patients should be engaged independently from, or together with, caregivers in the CPG development process. Forming an advisory board (AB) could also be useful. Research suggests it is important to engage relevant stakeholders early on and ask for their input often [27]. Working with a patient advocacy organization can help locate patients, caregivers, and others with relevant perspectives who can provide input on patient needs, the feasibility of proposed engagement activities, appropriate participation burden, and acceptable remuneration for participation. Patient representatives can be instrumental in helping operationalize the engagement tasks, define key concepts, translate scientific information, and finalize research protocols [28, 29]. All research-related activities should be reviewed and approved by the institutional review board.

Examples We worked with the Duchenne Registry to identify key patient and caregiver partners and assembled a multi-stakeholder AB that included one adult with DMD, two caregivers, two clinicians, two genetic counselors, three researchers, and two guideline developers.Footnote 1 The AB was co-led by a caregiver and a Delphi expert who made sure that all decisions were made jointly and that the patient/caregiver voices were heard, valued, and given more weight (than those of the other AB members) in discussions related to decisions that may have affected what the panelists were asked to do and how the panel results were interpreted. We found patient and caregiver input particularly useful for helping us define, measure, and operationalize patient-centeredness in the guideline context (see Box). Caregivers and patients on the AB also helped us identify the recommendations that may be of interest to patients and caregivers. To ensure participants understood the complex medical information, we developed plain language explanations of each recommendation. Here, patients/caregivers worked with clinicians to finalize these descriptions. Using AB input, we also included the clinical rationale for each care consideration, a description of the process for following the guidance, and other relevant information, such as treatment burden.

figure a

Mirror Methods Used for Expert and Stakeholder Engagement

One way to increase the scientific rigor and legitimacy of patient engagement in CPG development is to adapt the methods that clinical experts use to develop guidelines. Because CPG development is labor intensive and time consuming, it is crucial to ensure that participants do not feel overburdened [30]. Finding a balance between rigor and ease of participation is key.

Examples To mirror the methods clinicians used for the 2018 DMD care considerations [26], we began Round 1 by providing study participants with data we collected in Round 0 on the reasons for, and the barriers and facilitators associated with, seeking care. We then asked participants to rate the patient-centeredness of guideline recommendations (Fig. 3). This corresponded to the step of providing clinical experts with a literature review before asking them to rate the appropriateness of different treatments. We also adopted a three-round modified-Delphi format and used a nine-point rating scale, which mirrored the appropriateness and necessity scales that clinicians used to develop the 2018 DMD care considerations. Finally, we adopted the RAM approach to determine consensus [25].

Fig. 3
figure 3

Round 1

Full size image

Pilot Test the Engagement Approach

It is best practice to pilot test any data collection with a small sample of qualified participants [31]. A pilot is particularly important for online modified-Delphi approaches [32] because the task is novel for a typical patient and there are nuances to using online platforms. It is also important for ensuring participants can actually use the online tool, especially if they have disabilities. Guideline developers and panel participants are not in the same room and cannot provide assistance in real time. It is important to ensure pilot testers are not counted as study participants.

Examples Based on our experiences [33], we recommend testing the clarity of participation instructions, recommendation wording, and rating criteria. A pilot allowed us to estimate the time that participation in each round was likely to take, which helps determine the amount of renumeration, if any. Asking testers for feedback at the end of the pilot via a survey or brief telephone interview can help identify how the wording of recommendations should be changed, what information to add or delete, or how to improve the engagement process. Based on feedback we received during the pilot, we reduced the number of recommendations that participants had to rate.

Recruit Participants with Diverse Perspectives

Expert panels are often criticized for not including diverse perspectives. A panel about the clinical appropriateness of carotid endarterectomy that includes only surgeons will arrive at different recommendations than a panel of surgeons, neurologists, primary care physicians, and radiologists [34]. The same can be true of patient panels. It is important to ensure that patient representatives have relevant experiences and to help them think about the experiences of a typical patient, especially if patient-only panels use a methodology that clinical panels adhere to.

Examples We found that using an established and curated patient registry was helpful for recruiting a panel with diverse views. While it may be difficult to know what types of patients may have different views on a given issue, we were able to reach the diversity goal by using previous research on patient preferences, recruiting demographically and geographically diverse panelists, and recruiting those in different stages of disease progression. If recruitment via registries is not possible, then screening should be used to confirm a participant’s expertise with a condition.

Assemble a Panel of Adequate Size and Composition

Assembling panels of adequate size and composition helps ensure effective and productive online discussion and account for attrition in online modified-Delphi panels. Research suggests empaneling approximately 40 participants; larger panels may increase participation burden during the discussion round, and smaller panels may become too small due to attrition [35]. Attrition is typical for all Delphi panels because they rely on iterative data collection [36]. It is not uncommon for online Delphi panels with only two rating rounds to have 50% participation rates, calculated by dividing the number of those completing all rounds by the number of those invited to participate [37].

Examples To account for attrition, we included 61 participants in each panel. To reduce attrition, we asked participants during recruitment to confirm their interest and intention to participate. We made sure both panels consisted of patients and caregivers to ensure diversity of perspectives. Because DMD is a rare pediatric disorder, most participants were parents of, or caregivers to, individuals with DMD, but we also included adults with DMD.

Implementation and Continuous Participant Engagement

Build Participant Research and Engagement Capacity

CPG groups require patients and their representatives to undergo extensive training on the CPG development process, which can make patients unwilling to engage [22]. Although an online platform can help reduce perceived participation burden, it is important to ensure that participation instructions and task descriptions are self-explanatory. Because some participants are more comfortable with online technologies and sharing disease experiences, CPG developers should try to put all participants on a level playing field.

Examples To build their capacity, we provided participants with instructions on how to participate in the online process and use the online platform. The instructions were modified based on the pilot results. We included instructional videos on how to log into ExpertLens and participate in each round. Because Round 2 used charts showing the distribution of participants’ responses, we provided explanations of what each chart showed, included tooltips that explained statistical terms, and color-coded group responses/decisions (i.e., green text identified recommendations that participants agreed were important or acceptable) (Fig. 4). In case participants had questions or technical issues, they received contact information for study staff, including the principal investigator, caregiver representative, clinician, and technical support personnel.

Fig. 4
figure 4

Round 2

Full size image

Build Two-Way Interaction

Although face-to-face interaction may be more engaging than online discussion boards, threaded discussion boards allow participants to engage in more thoughtful conversations and explore other participants’ ideas [38]. That is why encouraging two-way information exchange and lively discussions is particularly important for online modified-Delphi panels. Make sure discussion boards have a clear structure and allow participants to keep track of comments made by other participants. As with in-person expert panels, an experienced discussion facilitator is crucial. The facilitator’s role is to encourage discussion, solicit comments from all participants, and ensure that no single participant dominates the conversation [25, 39].

Examples In our experiences, providing the distribution of Round 1 responses and a summary of participants’ rationales in Round 2 helps promote discussion because participants see how their responses compare with those of other participants. A threaded discussion board structure makes it easier for participants to find the right place to share their opinions (Fig. 4). Using participant IDs helps ensure that all comments made by a given participant can be attributed to him or her, and the anonymity facilitates an open exchange of information. We found it useful for the user ID to show whether a participant was a caregiver or a patient to help participants contextualize their comments [49].

To ensure active discussion engagement, three trained discussion moderators (a caregiver, a genetic counselor, and a modified-Delphi expert) facilitated the discussions by reviewing and posting comments at least once a day. Moderators followed a guide (see Appendix A) and were instructed to focus on group dynamics, ask non-leading clarifying questions, promote direct engagement among participants, and answer factual questions about the study. They also provided access to additional informational resources as needed.

Ensure Continuous Engagement and Retention of Participants

Because participant attrition is common in Delphi panels [32, 36], it is important to keep panelists engaged throughout all study rounds. The Delphi method is less common than surveys and relies on iterative data collection. Panelists can participate at any time while each round is open but are expected to contribute to each round. Because of the time gap between rounds, reminding them about their participation is critical.

Examples To encourage continuous engagement, we informed participants about expected time commitments and paid them $US50 for completing each round. We sent personalized email invitations when each round opened and emailed up to three reminders to lagging participants during each round. We extended the round deadlines as needed. If requested, we allowed participants to perform Round 1 after Round 2 opened but before they saw other participants’ responses and comments. Such flexibility may be required when the condition of interest causes significant impairment or treatment burden. During Round 2, participants also received daily discussion digests informing them of when others posted new comments or responded to the participant’s own comments.

Conduct Scientifically Rigorous Data Analysis

Research shows that the methods used to measure consensus can have a significant impact on study findings [40] and calls for specifying how Delphi data will be analyzed before they are collected [41]. The RAM manual offers a validated and frequently used measure of consensus for nine-point Likert scales [25]. Moreover, Delphi panels have been criticized for low replicability of its findings [42]. Therefore, it is prudent to conduct more than one panel using the same protocol, balance panel composition on key variables that might affect outcomes, and include data from all panels in the a priori determination of group consensus [43]. Because the Delphi technique is based on a mixed-methods approach to data collection, thematic analysis of qualitative comments can help explain why consensus was or was not reached [44].

Examples To ensure rigor of our panel findings, we published our research protocol at the beginning of the project [18] and used the RAM to measure consensus [25]. We also ran two concurrent panels using the same protocol to ensure replicability of panel findings. We randomly assigned selected participants to one of two panels and balanced panels in terms of caregiver educational attainment, ambulatory status of the individual with DMD, and the distance to the closest PPMD Certified Duchenne Care Center [45], which we considered key variables that might affect determinations of patient-centeredness [46]. Our a priori criteria for patient-centeredness was that both panels had to agree that a recommendation was important and acceptable. Finally, we qualitatively analyzed all comments made by participants throughout the panel to determine points of agreement and disagreement and any differences in perspectives between patients and caregivers.

Evaluation and Dissemination

Evaluate Engagement Activities

Participant experiences with the Delphi processes are not typically evaluated as part of every panel. Understanding what works and what does not is important for measuring the quality of panel findings and the engagement process as well as for retaining participants during iterative data collection [47].

Examples All panels conducted using the ExpertLens system include questions that measure participant experiences and satisfaction with the platform [48]. For our study, we slightly modified these questions and asked them after Rounds 1 and 3. We also interviewed a diverse sample of individuals with DMD and their caregivers after the modified-Delphi process was completed [49].

Disseminate Results

Sharing results with participants [50] is a key principle of participant-centered research [51], and sharing individual results and overall study findings can help enroll and retain participants in longitudinal projects [52, 53]. Disseminating study findings to wider audiences, including patients, caregivers, clinicians, and guideline developers, is important not only for the conduct of rigorous and transparent research but also for improving care quality and helping develop future guidelines [2, 54].

Examples Feedback on Round 1 results provided to participants can serve as an important incentive to participate and engage in Delphi panels. In Round 2 of our study, we not only provided statistical summaries of Round 1 ratings, but also thematically analyzed the reasons behind participant ratings. We also emailed copies of Round 2 discussion comments to participants who requested them after the panels were completed. We presented preliminary study findings to our panelists using a webinar format that has been posted on the PPMD’s YouTube channel (https://www.youtube.com/watch?v=aps_E08C4fg). To reach a wider audience, we presented our results at the annual PPMD and G-I-N conferences, as well as at the Centers for Disease Control and Prevention, which was responsible for developing the 2018 DMD care considerations. In addition, we gave a G-I-N webinar, which was recorded and posted on the G-I-N North America’s website (https://g-i-n.net/library/webinars/g-i-n-n-a-webinars/a-new-online-approach-to-engaging-patients-andcaregivers-in-guideline-development/?searchterm=khodyakov). Finally, we published the results in peer-reviewed journals [19, 46, 49].

Conclusions

The importance of involving patients, caregivers, and/or their representatives in the process of developing CPGs has been recognized by guideline developers. We offer 11 practical considerations for using online modified-Delphi approaches to facilitate large-scale engagement. While we used the examples from a recent study that engaged individuals with DMD and their caregivers in rating the patient-centeredness of already finalized DMD care considerations, online modified-Delphi approaches could be used to engage relevant stakeholders not only throughout but also beyond other stages of guideline development.

However, online engagement requires specialized resources and has its limitations. First, guideline developers need access to an online platform with survey, discussion, and analytic capabilities; patients need access to an internet-connected device. Second, not every patient may find the online experience to be as engaging as in-person meetings. Nonetheless, people quickly become accustomed to using technology in all aspects of their lives, which is likely to increase their comfort level with online engagement moving forward. Third, discussion moderators need skills to facilitate asynchronous discussion among panelists. Finally, although online engagement is intended to be intuitive, training on how to participate in data collection activities should be provided.

Online modified-Delphi approaches may not be appropriate for every engagement activity. In selecting the type of engagement, guideline developers should consider its purpose, engagement tasks, and participants. Given the relative novelty of this online method and the fact that we engaged patients and caregivers after the DMD care considerations were finalized, future research should focus on evaluating the impact of online engagement of patients and caregivers on the quality of and adherence to guideline recommendations. Nonetheless, we believe that online engagement is a promising approach for guideline developers to consider and should be added to the G-I-N PUBLIC Toolkit.

Notes

  1. Please note that some AB members represented more than one stakeholder group (i.e., a clinician and a guideline developer).

References

  1. Barham L. Public and patient involvement at the UK National Institute for Health and Clinical Excellence. Patient Patient Center Outcomes Res. 2011;4(1):1–10.

    Google Scholar 

  2. Boelens PG, Taylor C, Henning G, et al. Involving patients in a multidisciplinary European consensus process and in the development of a ‘patient summary of the consensus document for colon and rectal cancer care’. Patient Patient Center Outcomes Res. 2014;7(3):261–70.

    Google Scholar 

  3. Rashid A, Thomas V, Shaw T, Leng G. Patient and public involvement in the development of healthcare guidance: an overview of current methods and future challenges. Patient Patient Center Outcomes Res. 2017;10(3):277–82.

    Google Scholar 

  4. Armstrong MJ, Mullins CD, Gronseth GS, Gagliardi AR. Impact of patient involvement on clinical practice guideline development: a parallel group study. Impl Sci. 2018;13(1):55.

    Google Scholar 

  5. Légaré F, Boivin A, van der Weijden T, et al. Patient and public involvement in clinical practice guidelines: a knowledge synthesis of existing programs. Med Decis Making. 2011;31(6):E45–74.

    PubMed  Google Scholar 

  6. Young A, Menon D, Street J, Al-Hertani W, Stafinski T. Engagement of Canadian patients with rare diseases and their families in the lifecycle of therapy: a qualitative study. Patient Patient Center Outcomes Res. 2018;11(3):353–9.

    Google Scholar 

  7. Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E. Clinical practice guidelines we can trust. Washington, DC: National Academies Press; 2011.

    Google Scholar 

  8. Qaseem A, Forland F, Macbeth F, OllenschlÃĪger G, Phillips S, van der Wees P. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156(7):525–31.

    PubMed  Google Scholar 

  9. Jarrett L, Patient Involvement Unit. A report on a study to evaluate patient/carer membership of the first NICE Guideline Development Groups. London: National Institute for Health and Care Excellence; 2004.

  10. Selva A, Sanabria AJ, Pequeno S, et al. Incorporating patients’ views in guideline development: a systematic review of guidance documents. J Clin Epidemiol. 2017;88:102–12.

    PubMed  Google Scholar 

  11. Coulter I, Elfenbaum P, Jain S, Jonas W. SEaRCH™ expert panel process: streamlining the link between evidence and practice. BMC Res Notes. 2016;9(1):16.

    PubMed  PubMed Central  Google Scholar 

  12. Winkler J, Moser R. Biases in future-oriented Delphi studies: a cognitive perspective. Technol Forecast Soc Chang. 2016;105:63–76.

    Google Scholar 

  13. Grant S, Hazlewood GS, Peay HL, et al. Practical considerations for using online methods to engage patients in guideline development. Patient Patient Center Outcomes Res. 2018;11(2):155–66.

    Google Scholar 

  14. Roman BR, Feingold J. Patient-centered guideline development: best practices can improve the quality and impact of guidelines. Otolaryngol Head Neck Surg. 2014;151(4):530–2.

    PubMed  Google Scholar 

  15. Atkins D, Perez-Padilla R, MacNee W, Buist AS, Cruz AA. Priority setting in guideline development: article 2 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):225–228.

  16. den Breejen EM, Nelen WL, Knijnenburg JM, Burgers JS, Hermens RP, Kremer JA. Feasibility of a wiki as a participatory tool for patients in clinical guideline development. J Med Internet Res. 2012;14(5):e138.

    Google Scholar 

  17. Lamontagne M-E, Perreault K, Gagnon M-P. Evaluation of the acceptability, feasibility and effectiveness of two methods of involving patients with disability in developing clinical guidelines: study protocol of a randomized pragmatic pilot trial. Trials. 2014;15(1):118.

    PubMed  PubMed Central  Google Scholar 

  18. Khodyakov D, Kinnett K, Grant S, et al. Engaging patients and caregivers managing rare diseases to improve the methods of clinical guideline development: A research protocol. JMIR Res Protoc. 2017;6(4).

    PubMed  PubMed Central  Google Scholar 

  19. Khodyakov D, Denger B, Grant S, et al. The RAND/PPMD patient-centeredness method: A novel online approach to engaging patients and their representatives in guideline development. Eur J Person Center Healthc (in Press).

  20. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476.

    CAS  PubMed  PubMed Central  Google Scholar 

  21. Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2):e1000217.

    PubMed  PubMed Central  Google Scholar 

  22. Boivin A, Currie K, Fervers B, et al. Patient and public involvement in clinical guidelines: international experiences and future perspectives. Qual Saf Health Care. 2010;19(5):1–4.

    Google Scholar 

  23. Brady SR. Utilizing and adapting the Delphi method for use in qualitative research. Int J Qual Methods. 2015;14(5):1609406915621381.

    Google Scholar 

  24. Hasson F, Keeney S, BA M, McKenna H, Adv Dip Ed R. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008.

    CAS  PubMed  Google Scholar 

  25. Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA appropriateness method user’s manual. Santa Monica: RAND Corporation; 2001.

    Google Scholar 

  26. Birnkrant DJ, Bushby K, Bann CM, et al. Diagnosis and management of Duchenne muscular dystrophy, part 1: diagnosis, and neuromuscular, rehabilitation, endocrine, and gastrointestinal and nutritional management. Lancet Neurol. 2018;17(3):251–67.

    PubMed  PubMed Central  Google Scholar 

  27. Forsythe LP, Ellis LE, Edmundson L, et al. Patient and stakeholder engagement in the PCORI pilot projects: description and lessons learned. J Gen Intern Med. 2016;31(1):13–21.

    PubMed  Google Scholar 

  28. Koay PP, Sharp RR. The role of patient advocacy organizations in shaping genomic science. Annu Rev Genom Hum Genet. 2013;14:579–95.

    CAS  Google Scholar 

  29. Wong-Rieger D. Moving from patient advocacy to partnership: a long and bumpy road. Patient Patient Center Outcomes Res. 2017;10(3):271–6.

    Google Scholar 

  30. Cluzeau F, Wedzicha JA, Kelson M, et al. Stakeholder involvement: how to do it right: article 9 in integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):269–273.

    PubMed  Google Scholar 

  31. Clibbens N, Walters S, Baird W. Delphi research: issues raised by a pilot study. Nurse Res. 2012;19(2):37–44.

    PubMed  Google Scholar 

  32. Keeney S, McKenna H, Hasson F. The Delphi technique in nursing and health research. Wiley, Oxford; 2010.

  33. Khodyakov D, Grant S, Barber CEH, Marshall DA, Esdaile JM, Lacaille D. Acceptability of an online modified Delphi panel approach for developing health services performance measures: results from 3 panels on arthritis research. J Eval Clin Pract. 2017;23(2):354–60.

    PubMed  Google Scholar 

  34. Leape LL, Park RE, Kahan JP, Brook RH. Group judgments of appropriateness: the effect of panel composition. Int J Qual Health Care. 1992;4(2):151–9.

    CAS  Google Scholar 

  35. Khodyakov D, Hempel S, Rubenstein L, et al. Conducting online expert panels: a feasibility and experimental replicability study. BMC Med Res Methodol. 2011;11(1):174.

    PubMed  PubMed Central  Google Scholar 

  36. van Teijlingen E, Pitchforth E, Bishop C, Russell E. Delphi method and nominal group technique in family planning and reproductive health research. J Fam Plan Reprod Health Care. 2006;32(4):249–52.

    Google Scholar 

  37. Elwyn G, O’Connor A, Stacey D, et al. Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. BMJ. 2006;333(7565):417.

    PubMed  PubMed Central  Google Scholar 

  38. Meyer KA. Face-to-face versus threaded discussions: the role of time and higher-order thinking. J Asynchron Learn Netw. 2003;7(3):55–65.

    Google Scholar 

  39. Handels RL, Wolfs CA, Aalten P, et al. Optimizing the use of expert panel reference diagnoses in diagnostic studies of multidimensional syndromes. BMC Neurol. 2014;14(1):190.

    PubMed  PubMed Central  Google Scholar 

  40. Grant S, Booth M, Khodyakov D. Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies. J Clin Epidemiol. 2018;99:96–105.

    PubMed  Google Scholar 

  41. Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. Am J Public Health. 1984;74(9):979–83.

    CAS  PubMed  PubMed Central  Google Scholar 

  42. Shekelle P, Kahan J, Bernstein S, Leape L, Kamberg C, Park R. The reproducibility of a method to identify the overuse and underuse of medical procedures. N Engl J Med. 1998;338(26):1888–95.

    CAS  PubMed  Google Scholar 

  43. Khodyakov D, Mikesell L, Schraiber R, Booth M, Bromley E. On using ethical principles of community-engaged research in translational science. Transl Res. 2016;171(52–62):e51.

    Google Scholar 

  44. Dalal SR, Khodyakov D, Srinivasan R, Straus SG, Adams J. ExpertLens: a system for eliciting opinions from a large pool of non-collocated experts with diverse knowledge. Technol Forecast Soc Chang. 2011;78(8):1426–44.

    Google Scholar 

  45. Kinnett K, Dowling JJ, Mendell JR. The certified duchenne care center program. Neuromuscul Disord. 2016;26(12):853–9.

    PubMed  Google Scholar 

  46. Denger B, Kinnett K, Martin A, Grant S, Armstrong C, Khodyakov D (2019). Patient and caregiver perspectives on guideline adherence: the case of endocrine and bone health recommendations for Duchenne muscular dystrophy. Orphanet J Rare Dis. 14:205. DOI: 10.1186/s13023-019-1173-7

    Article  PubMed  PubMed Central  Google Scholar 

  47. Hall DA, Smith H, Heffernan E, Fackrell K, for the Core Outcome Measures in Tinnitus International Delphi Research Steering G. Recruiting and retaining participants in e-Delphi surveys for core outcome set development: Evaluating the COMiT’ID study. PLOS One. 2018;13(7):e0201378.

    PubMed  PubMed Central  Google Scholar 

  48. Khodyakov D, Grant S, Meeker D, Booth M, Pacheco-Santivanez N, Kim KK. Comparative analysis of stakeholder experiences with an online approach to prioritizing patient-centered research topics. J Am Med Inform Assoc. 2016;24(3):537–43.

    Google Scholar 

  49. Armstrong C, Grant S, Kinnett K, et al. Participant experiences with a new online modified-Delphi approach for engaging patients and caregivers in developing clinical guidelines. Eur J Person Center Healthc (in press).

  50. National Academies of Sciences E, and Medicine,. Returning individual research results to participants: guidance for a new research paradigm. Washington, DC: The National Academies Press; 2018.

  51. Aungst J, Haas A, Ommaya A, Green LW. Exploring challenges, progress, and new models for engaging the public in the clinical research enterprise: clinical research roundtable workshop summary. Washington, DC: National Academies Press; 2003.

    Google Scholar 

  52. Khodyakov D, Mendoza -G, Alexandra, Berry S, Nebeker C, Bromley E. Return of value in the new era of biomedical research—one size will not fit all. AJOB Emp Bioethics (in press).

  53. Khodyakov D, Bromley E, Evans SK, Sieck K. Best Practices for Participant and Stakeholder Engagement in the All of Us Research Program. 2018; https://www.rand.org/pubs/research_reports/RR2578.html.

    Google Scholar 

  54. Dagenais S, Tricco AC, Haldeman S. Synthesis of recommendations for the assessment and management of low back pain from recent clinical practice guidelines. Spine J. 2010;10(6):514–29.

    PubMed  Google Scholar 

Download references

Acknowledgements

The authors are grateful to Ben Dupree, Joanna Johnson, Glen Hazlewood, and Natalie Street for their input and guidance throughout the process of developing this paper. We thank all individuals with DMD and their caregivers who helped us test the new method for engaging patients in guideline development. We greatly appreciate Melissa Bauman’s editorial assistance and Courtney Armstrong for help with project management.

Author information

Authors and Affiliations

  1. RAND Health Care, RAND Corporation, 1776 Main Street, PO Box 2138, Santa Monica, CA, USA

    Dmitry Khodyakov, Sean Grant & Ian Coulter

  2. Parent Project Muscular Dystrophy, Hackensack, NJ, USA

    Brian Denger, Kathi Kinnett & Ann Martin

  3. RTI International, Research Triangle Park, NC, USA

    Holly Peay

  4. Indiana University, Richard M. Fairbanks School of Public Health, Indianapolis, IN, USA

    Sean Grant

Authors
  1. Dmitry Khodyakov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sean Grant
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Brian Denger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kathi Kinnett
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ann Martin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Holly Peay
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ian Coulter
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

DK conceived the idea for this study and led the process of conducting all research activities. All co-authors contributed to the design and planning of the study. DK wrote the first draft of the manuscript. All co-authors reviewed, provided critical revisions for, and approved the final version of the manuscript.

Corresponding author

Correspondence to Dmitry Khodyakov.

Ethics declarations

Conflicts of interest

Dmitry Khodyakov is a leader of the ExpertLens team at RAND. Sean Grant is a member of the ExpertLens team. His spouse is a salaried employee of and owns stock in Eli Lilly and Company. Sean Grant has accompanied his spouse on company-sponsored travel. Brian Denger, Kathi Kinnett, Ann Martin, Holly Peay, and Ian Coulter have no conflicts of interest that are directly relevant to the content of this review/study.

Funding

This work was supported through a Patient-Centered Outcomes Research Institute (PCORI) Program Award (ME-1507-31052). All statements in this article, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of PCORI, its Board of Governors or Methodology Committee or the official position of the Centers for Disease Control and Prevention.

Appendix A: ExpertLens Moderator Protocol

Appendix A: ExpertLens Moderator Protocol

Rationale for the Moderator

ExpertLens is a process that allows participants to consider, individually and as a group, options for setting priorities related to a specific topic or explore agreement among the group members on a given issue. Like most deliberative processes, the goal is for participants to make decisions based on a combination of their personal or professional experience; accurate information; their personal and societal values; and the exchanges they have with others who relate different perspectives and experiences.

To encourage participant engagement using internet-based asynchronous discussions, interaction among participants will be facilitated by a discussion moderator. To improve participant experiences and to achieve the highest level of participant engagement, the moderator should encourage participant interaction; suggest new discussion topics that expand the depth and breadth of the existing discussion topics; stimulate greater engagement in the topic; and encourage individuals’ active discourse and critical thinking.

Important Features of the Moderator’s Role

  • Is non-directive: The moderator is not the “discussion leader,” but rather shadows and encourages the conversations, intervening when needed to maximize the experience for participants and improve the potential for results to be truly based on deliberation.

    Example: No one has commented on Question 2. Does anyone have thoughts about Round 1 responses?

  • Remains neutral: The moderator asks questions, makes observations, probes for reactions—but remains neutral on all content under discussion (i.e., the moderator’s opinion is never evident to others).

    Example: Participant 01 seems to be making this point. What do others think about this?

  • Points out differing views: Because the purpose of Round 2 is to debate different perspectives, it is especially important to reflect on the group’s responses when there are meaningful differences or controversies. However, consensus in a diverse group is something that should also be acknowledged.

    Example: It appears Participant 01 thinks this, while Participant 02 thinks this. What views do others have to say about it?

  • Intervenes selectively: Moderator only contributes when there is a good reason to (e.g., when the discussion process is going too slowly, drags on too long, changes to an irrelevant topic or is missing an important issue to discuss).

    Example: No one seemed to answer Participant 02’s question: does anyone have any thoughts?

  • Plays devil’s advocate: If agreement on a topic comes too easily or quickly, it is useful for the moderator to encourage participants to think more broadly about the issue. But this must be done without suggesting it is the moderator’s view.

    Example: I’ve heard other people take the opposite view that relying on patients to be responsible for their own health doesn’t work. How would you respond to those who say that? This technique can also be used if a participant presents information that is clearly inaccurate. It’s best to let others “correct” the fact, but if a correction is not forthcoming, moderator can use “I’ve heard others say…” Correcting participants, however, should be done sparingly and only if wrong information is really a problem for the discussion.

  • Recaps to simplify: If introducing a comment or question in the middle of a discussion, give a brief summary of what the discussion is before posting. This saves people time, and they may be more likely to respond if it is clear from one reading what is being debated.

    Example: Comments so far seem to suggest … Do others agree, or do you have different opinions?

  • Varies the postings: Not all moderator contributions should be questions. Other posting types could be summaries of long discussions or observations about the sides of an ongoing debate. Too many questions may start looking like a quiz.

    Example: It is interesting that the following two points have been made:…

What Should the Moderator Do During ExpertLens?

When Round 1 is ending:

  • Look at the distribution of participants’ responses: what is the pattern of those responses? (e.g., are there two dominant reactions? Are responses widely scattered? Is there an obvious minority view? Do participants have similar responses to rating of the criteria?)

  • Review rationale comments provided by participants in Round 1 to explain their numeric responses to rating questions.

  • Based on these observations, prepare specific probes that bring these initial results to participants’ attention.

  • Example: It appears that this group has two very different reactions to the first research study listed. Could someone offer a more detailed explanation on why it is rated so low on most of the criteria?

  • Post some discussion comments BEFORE Round 2 opens. This can make it easier for participants to engage with the discussion board.

During Round 2:

  • Give participants enough time to respond to your first comment.

  • If conversation is lagging, post a comment that is more specific to how some participants voted—this makes it harder for them to ignore.

  • Example: Several of you felt strongly that research topic #3 (describe) is much more likely to achieve the aspirational goal. What is it about this one that makes you optimistic?

  • Log into ExpertLens system at least once a day during Round 2 and post comments. This should not necessarily be new discussion threads. It can be responses to already posted threads.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Khodyakov, D., Grant, S., Denger, B. et al. Practical Considerations in Using Online Modified-Delphi Approaches to Engage Patients and Other Stakeholders in Clinical Practice Guideline Development. Patient 13, 11–21 (2020). https://doi.org/10.1007/s40271-019-00389-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40271-019-00389-4