Journal of General Internal Medicine

, Volume 32, Issue 8, pp 883–890 | Cite as

Engaging Stakeholders to Inform Clinical Practice Guidelines That Address Multiple Chronic Conditions

  • Wendy L. Bennett
  • Craig W. Robbins
  • Elizabeth A. Bayliss
  • Renee Wilson
  • Heather Tabano
  • Richard A. Mularski
  • Wiley V. Chan
  • Milo Puhan
  • Tsung Yu
  • Bruce Leff
  • Tianjing Li
  • Kay Dickersin
  • Carol Glover
  • Katie Maslow
  • Karen Armacost
  • Suzanne Mintz
  • Cynthia M. Boyd
Original Research



Having more than one chronic condition is common and is associated with greater health care utilization, higher medication burden and complexity of treatment. However, clinical practice guidelines (CPGs) do not routinely address the balance between harms and benefits of treatments for people with multiple chronic conditions (MCCs).


To partner with the Kaiser Permanente Integrated Cardiovascular Health (ICVH) program to engage multiple stakeholders in a mixed-methods approach in order to: 1) identify two high-priority clinical questions related to MCCs, and 2) understand patients’ and family caregivers’ perceptions of meaningful outcomes to inform benefit/harm assessments for these two high-priority questions. These clinical questions and outcomes will be used to inform CPG recommendations for people with MCCs.

Design and Participants

The ICVH program provided 130 topics rank-ordered by the potential for finding evidence that would change clinical recommendations regarding the topic. We used a modified Delphi method to identify and reword topics into questions relevant to people with MCCs. We used two sets of focus groups (n = 27) to elicit patient and caregiver perspectives on two important research questions and relevant patient-important outcomes on benefit/harm balance for people with MCCs.

Key Results

Co-investigators, patients and caregivers identified “optimal blood pressure goals” and “diabetes medication management” as important clinical topics for CPGs related to people with MCCs. Stakeholders identified a list of relevant outcomes to be addressed in future CPG development including 1) physical function and energy, 2) emotional health and well-being, 3) avoidance of treatment burden, side effects and risks, 4) interaction with providers and health care system, and 5) prevention of adverse long-term health outcomes.


Through the application of a mixed-methods process, we identified the questions regarding optimal blood pressure goals and diabetes medication management, along with related patient-centered outcomes, to inform novel evidence syntheses for those with MCCs. This study provides the lessons learned and a generalizable process for CPG developers to engage patient and caregivers in priority-setting for the translation of evidence into future CPGs. Ultimately, engaging patient and stakeholders around MCCs could improve the relevance of CPGs for the care of people with MCCs.


qualitative research multiple chronic conditions stakeholder engagement 


Forty-eight percent of people older than age 65 have three or more chronic health conditions.1 , 2 Having multiple chronic conditions (MCCs) increases costs, risk of polypharmacy and the complexity of treatment regimens.3 Understanding the potential overlap between various chronic health conditions is especially important as each condition may influence the care of the other condition(s) through limitations of life expectancy, interactions between therapies and/or direct contraindications to therapy.4 , 5 Since most clinical practice guidelines (CPGs) focus on a single disease, people with MCCs may experience disease-focused—and thus, by definition, not patient-centered—care.3 , 6 10 Ideally, the balance between treatment benefits and harms or side effects of any intervention among people with MCCs would and should be reflected in CPGs.3 , 6 10

Because CPGs have the potential to influence clinical practice, improve quality of care and reduce the inappropriate use of tests and therapies,11 the Institute of Medicine (IOM) has suggested standards to enhance their validity, trustworthiness and rigor.12 As part of these standards, the IOM recognized the importance of engaging patients and other stakeholders in the development of CPGs, through the formulation of clinical questions and external review of the final documents, and encouraged strategies to foster increased participation throughout the entire process.12 Guideline development organizations need effective and efficient methods for engaging patients in this process.13 The Patient-Centered Outcomes Research Institute (PCORI) has recognized both the necessity and associated challenges with engaging patients as stakeholders and co-investigators in research and CPG development.14 , 15 This study describes the first aim of a larger project involving a partnership with the Kaiser Permanente National Guideline Program to set priorities for clinical topics and outcomes relevant to people with MCCs. In 2010, Kaiser Permanente’s National Guideline Program reorganized its guideline work to bring recommendations for diabetes, hypertension, hyperlipidemia, coronary artery disease and heart failure under the scope of one guideline program (thereafter called the Integrated Cardiovascular Health [ICVH] program). Given this reorganization to address concordant conditions, the project team determined this ICVH guideline program to be best suited to serve as a platform for a methods project addressing the broader landscape of MCCs beyond cardiovascular health.16 We sought to build on best practices for successful and effective stakeholder engagement in research and to apply these lessons learned to engage multiple stakeholders in the process of evidence synthesis to inform CPG development for people with MCCs. Findings from this effort are needed to directly inform the subsequent aims of this project in carrying out a quantitative benefit/harm assessment to inform Kaiser Permanente’s CPG development. Ultimately, the overarching goal of this project is to inform recommendations and person-centered care for people with MCCs.

We used a multi-step mixed-methods process to engage multiple stakeholders in order to accomplish the following study objectives: 1) to identify two high-priority questions related to MCCs, and 2) to understand patients’ perceptions of meaningful outcomes to inform guideline-driven treatment decisions for these two high-priority questions.


The institutional review boards of Johns Hopkins University and Kaiser Permanente Colorado approved this study.

Identification of Guideline Topics for Addressing MCCs

The Kaiser Permanente National Guideline Program provides evidence-based recommendations to its organizations that support the delivery of care and optimize the health of its patients. Each year, the ICVH clinical leads review and prioritize the existing recommendations in need of updating based on 1) the importance of maintaining a recommendation on a clinical topic, and 2) the likelihood of needing to update the literature search and recommendation because of new evidence. The ICVH clinical program provided the research team with their list of high-priority topics for 2014. This topic list had been generated by clinicians and guideline developers and served as the starting point for the identification of high-priority topics related to the presence of these and other chronic conditions.

Overview of Steps for Topic Prioritization and Opportunities for Stakeholder Engagement

We used a six-step process for multi-stakeholder prioritization of cardiovascular health-related topics and outcomes important to people living with MCCs. At each stage of our prioritization, we engaged the following four stakeholder groups: 1) patient and caregiver co- investigators, 2) co-investigators who were leaders from the Kaiser Permanente National Guideline Program, 3) co-investigators who also served as clinical leads for the ICVH Guideline Program, and 4) patients and caregivers with MCCs from Kaiser Permanente Colorado. Topic prioritization in steps 1 through 4 involved patient and caregiver co-investigators and Kaiser Permanente National Guideline Program co-investigators. Topic prioritization and outcome generation in steps 5 and 6 involved focus groups with patients and caregivers with MCCs. We describe each of the steps below and in Figure 1.
Figure 1

Steps for topic and outcome prioritization with stakeholder engagement.

Steps 1 and 2: Identification of Topics of Clinical Importance for Adults with MCCs

In the first step, the research team eliminated topics not relevant for MCCs, regarding screening, pediatric patients/conditions and pregnant women. We then excluded the bottom tertile of topics, because they were deemed lowest priority and not likely to be updated (Fig. 1). Using two Delphi rounds,17 our entire team, including CPG developers and patient and caregiver co-investigators, reviewed the guideline topics and responded to the queries: “Is the question relevant to people with multiple chronic conditions, or can it be modified to be relevant?” (Yes, No, or Yes if modified). We ranked the “overall relevance to people with MCCs after modification” (scale: 1 = least relevant, 9 = most relevant). Questions with scores below the calculated mean were eliminated.

Steps 3 and 4: Creation of Broader Clinical Categories of Questions, Patient-Friendly Language

For steps 3 and 4, the research team organized the list of questions into broader categories of questions (e.g. hypertension, diabetes mellitus, hypercholesterolemia, coronary artery disease) and rephrased the remaining 12 questions into patient-friendly language with feedback from our patient and caregiver co-investigators. The principal investigator (PI) (CB) presented these questions to a meeting of the ICVH clinical leads to get their feedback about the relevance of these questions to their needs and to patients with MCCs.

Steps 5 and 6: Focus Groups with Patients and Caregivers

In step 5 we conducted two focus groups with patients and caregivers to elicit patients’ perspectives and to select two high-priority questions relevant to living with or caring for loved ones with MCCs. We had determined a priori the need to choose two questions to ensure that the next steps of this project were feasible and practical. Following step 5, in step 6 we conducted two more focus group with patients and caregivers, focused on identifying outcomes most important to patients with MCCs related to the two questions identified in step 5.

Conduct of Focus Groups in Steps 5 and 6 in the Mixed-Methods Process (Fig. 1)

Our team developed a semi-structured moderator guide for each set of focus groups (see Supplement), pilot-tested the guides with our patient and family caregiver co-investigators, and incorporated their feedback. In step 5, focus group participants received large printed cards with the health condition question and definitions. These cards were used to facilitate further discussion and the selection of high-priority topics. The moderator guide for step 6 utilized more open-ended questions and prompts related broadly to patient-important outcomes and treatment goals. Moderators from our clinical team used the nominal group technique in focus groups to enable in-depth discussions in both steps 5 and 6.17

Participants and Setting for Focus Groups in Steps 5 and 6

We identified focus group participants as eligible if they were members of Kaiser Permanente Colorado and had 1) at least one cardiovascular condition (coronary artery disease, hypertension, heart failure, hyperlipidemia, diabetes mellitus, stroke, transient ischemic attack and peripheral arterial disease/peripheral vascular disease), 2) one or more comorbidities from those listed in the Quan adaptation of the Elixhauser comorbidity index,18 and 3) a score of 3 or more based on the Quan adaptation of the Elixhauser comorbidity index used as a marker of disease severity18 to identify focus group participants likely to have MCCs. We also enriched our sample by including patients with dementia and invited family members or caregivers of participants. Focus groups lasted approximately 90 min and were held in convenient settings at Kaiser Permanente facilities in Denver, CO, in July and September 2015.

Qualitative Analysis of Focus Groups in Steps 5 and 6

The four focus groups were audiotaped and transcribed verbatim, and references to personal health information were removed. To develop the coding templates, three investigators (RW, HT, WB) independently read the transcript from the first group of each set, applying an editing style analysis to identify meaningful segments and create codes.19 Two investigators then independently coded the remaining transcripts, applying the coding template, which was iteratively modified as the analysis proceeded, and grouped codes into general themes. The team, including the patient-caregiver co-investigators, collectively selected the themes and representative quotes we present in this paper. To facilitate data management and coding, we used ATLAS.ti 6.2 software (ATLAS.ti Scientific Development GmbH, Berlin, Germany).


Flow Through the 6-Step Multiple Stakeholder Engagement Process (Fig. 1)

Figure 1 shows the flow through the steps in the stakeholder engagement and prioritization process. In step 1, the guideline program provided the research team with a list of 130 topics in rank order. Examples of high-priority topics were “anti-platelet therapy following stent placement…,” “medication management for type 2 diabetes” and “treatment for heart failure with preserved ejection fraction.” We eliminated the four topics addressing screening or primary prevention, pediatric populations and pregnancy, and the 42 topics in the bottom tertile, yielding 84 topics. Step 2 involved two Delphi rounds where 72 of the 84 questions were modified by investigators to make them more relevant to people with MCCs, and 52 questions were eliminated as part of the Delphi process. In step 3, the remaining 32 topics were organized into broader categories by disease condition, and were narrowed and reworded into 12 patient-friendly questions in step 4 (Fig. 1).

Steps 5 and 6 informed the question and outcomes selection by engaging patients with MCCs and their caregivers in four focus groups. Step 5 involved a total of 11 participants selecting from questions important for patients and caregivers with MCCs. Step 6 involved a total of 16 participants discussing high-priority outcomes for people with MCCs.

Results of Step 5: Focus Groups Selected Two High-Priority Questions Relevant to People with MCCs (Fig. 1, Table 1)

Table 1

Topics and Prioritization Ranking During First Steps of Focus Groups


Illustrative quotations

Blood pressure management

 Choosing of blood pressure treatments

I think it’s nuts…. I did have a medication once that caused me to cough…That’s the only time I’ve ever gone to somebody and said, “Let’s change this.”

Certain medications you add differently to the individual.

 Targeting blood pressure levels

I was on the smallest dose of blood pressure medicine, but I started working out several times per week, and my blood pressure was dropping lower. [I]t wasn’t within the standard guidelines goals, but that small dose was impacting me. So my doctor finally took me off blood pressure medication.

Diabetes management

 Choice of diabetes treatments

If you are borderline [diabetic, you might not take medication, but] some people take medicine and some people take shots, so there must be a reason why you can take a pill and somebody else have to take insulin.

I am a borderline diabetic. I don’t take medication. I exercise and, you know, they told me to do these things to keep it down. But I shouldn’t be on the same as he would be. Or his brother.

And I shouldn’t be taking their type medication.

 Goals for blood sugar level

And you wonder about the A1c. Everybody is saying it has to be below 7. But what about 7.2 being fine? But these pharmaceutical companies want the money.

Cholesterol management

 Figuring out who needs cholesterol medicine

My cholesterol was always high …my doctor always says it’s ok ‘cause it’s my good cholesterol that is pushing the ratio up…I didn’t need any medication…After I saw her…I got a letter …saying your cholesterol is high…and you better learn how to control it and all this stuff.

If you have high blood pressure and high cholesterol—if you are treating one of them, is the benefit from, like, one of them going to lower …going to give you added benefit from the other one without actually taking another medication? Do I have to take both in tandem? Or is there one that is going to treat both?

 Treatments for cholesterol/goals of treatment

You know what I’d like to see? I’d like to see this percentage of the people who take this medication and suffer from muscle aches or fatigue or blah, blah, blah, instead of the long list on that little piece of paper you get…

Coronary artery disease

 Blood thinners

They treat you like guinea pigs to see what works. That’s what they do…this doesn’t work? Try this one. They just keep switching.

With the blood thinner, I think it would be very important with regards to if a person was going to have surgery, to be very clear and find out if the person was taking something.

 Other treatments

…for me, the beta blocker, it really was like somebody sitting on my chest 24 h a day. But it was helping my heart at the same time. But I just could not function. So we changed. He said, ok, we’ll work in another way…but I said no more of them, because …once they found out I had COPD and that’s why I was going around sounding like a horse, you know?

Table 1 shows representative quotations for the 12 questions on the health conditions discussed in the first two focus groups (step 5): hypertension management, diabetes treatment, cholesterol treatment and management of coronary artery disease. Focus group participants shared their personal challenges in making decisions about medication choices and whether to take medications, given their other diseases and treatments. The two most commonly discussed questions as areas of great uncertainty and high relevance for people with MCCs were: “Of people with diabetes, are there groups of people for whom the balance between benefits and harms would affect the choice of treatment?” and “Should target blood pressure be different for some groups of people?”. One participant summarized his rationale for choosing diabetes as an important topic for people with MCCs: “I picked [the question on diabetes medications] as my first choice because diabetes is such a horrible disease, and I don’t think that the same treatment works for everybody…I think that is very important.” The two questions on blood pressure targets and diabetes medications were then promoted to step 6, for two more focus groups to identify high-priority outcomes.

Results of Step 6: Focus Groups Identified Outcomes Relevant to People with MCCs (Fig. 1, Table 2)

Table 2

Identification of High-Priority Outcomes on Diabetes and Hypertension Management in People with MCCs, Using Focus Groups



Physical function and energy

Physical function (activity, mobility)

Being able to do the things you enjoy or want to do

Comfort and overall feeling good

Having energy

Emotional health and well-being




Anger and guilt


Thinking clearly

Interaction with providers and health care system

Communication between patient and providers

Care coordination between providers

Advocating for self and own needs

Navigation of insurance and coverage benefits

Avoidance of treatment burden, risks and side effects

Medication burden

Drug interactions

Desire for relief without risks

Side effects of medications

Avoiding taking more medications

Being able to try alternative and holistic medicines

Using diet as a treatment

Utility of over-the-counter or alternative meds

Costs (co-pays, premiums) of medications and treatment

Role of genetics in choosing treatments

Prevention of adverse long-term health outcomes

Prevention of pain

Healthy aging

Kidney disease



Blood pressure level

Medication dependence/addiction

Surgery Avoidance

Eye complications: glaucoma/blindness

In step 6, we conducted two additional focus groups with patients with MCCs and their caregivers to discuss outcomes related to the questions on optimal blood pressure goals and diabetes medication management. Participants highlighted five broad outcome categories, as listed in Table 2 and described below, with examples of quotations: 1) physical function and energy, 2) emotional health and well-being, 3) avoidance of treatment burden, side effects and risks, 4) interaction with providers and the health care system, and 5) prevention of adverse long-term health outcomes.

Physical Function, Energy, Emotional Well-Being

Patients and caregivers valued outcomes of physical function, energy, quality of life and emotional well-being. They described chronic pain as a major barrier to day-to-day physical functioning and wellness, as exemplified by one participant who stated, “All these chronic pains that you have and then life knocks on the door, you have to be able to function.” Participants described their own experiences with aging, setting new health goals and addressing emotional health and well-being. One participant described his struggle to continue to function and maintain high quality of life:

My philosophy—and my wife and I fight about this all the time—I work out 6 days a week. … I take my dog out for walks even though I hurt so bad I can’t even begin to tell you how much pain I’m in when I’m walking. But I do it because I’m not going to let this thing stop me from living my life. I would rather only live for another 2 years and enjoy it, than to start watching everything I eat, and worry about this and worry about that, to live another 20 years.

Avoidance of Treatment Burden, Side Effects and Risks

Patients and caregivers highlighted outcomes of treatment burden, side effects (e.g. feeling “dizzy, falling out on the floor”) and the potential for drug interactions, especially when on multiple medications (“how [new medicines] affect everything else…[i]ncluding their health conditions and medications”). One participant with MCCs described her experience, weighing the potential benefits of taking multiple medications against her concerns for potential side effects and interactions that could lead to more health problems:

[T]o me, taking all the medications that are recommended for all of these chronic illnesses we all have—the side effects outweigh the possibility of recovery. Because the side effects are 16 pages long for this one little thing you are on…so you open the door to a whole bunch of other things you didn’t even have.

Participants commonly articulated the important outcome of assessing the burden of taking many different medications for different conditions: “I take nine or ten medications—different ones—twice a day. And I have anxiety, dementia and diabetes…just to name a few, and I take meds for all of that.”

Interaction with Providers and Health Care System

Participants valued communication and care coordination with health care providers as an outcome of their care. They expressed the need for providers to understand their unique situations and to “match” treatments to individual patient needs, prior history, having MCCs, race/ethnicity and personal/home situation. For example, two participants described their related questions and experiences:

Do you [have different recommendations] for different ethnic groups, like Native Americans, and different races? ‘Cause I’m American Indian, and diabetes is really rampant. [My father] never lost anything—toes or anything—but it shut everything down all at once…lungs, kidneys, heart…Do you [have different] targets for blood sugars for different races instead of just [one goal] for all?

My [cholesterol] prescription didn’t match me. So I don’t take it. I don’t take any. And it is high. It’s too high.

Participants also discussed their own strategies for how they go about making treatment decisions while weighing “risks and benefits” and “incorporating their individual needs” into their treatment choices. Some participants relied on their providers to know what was best for them:

I believe in my doctor and I do what he says, because he is the one who is taking care of me and he seems to know… And I can tell him, "This isn’t working. Do you have another remedy?”

Other participants were concerned that providers did not have the tools to make treatment decisions and desired greater autonomy. One participant described his process for weighing risks and benefits:

You weigh [the options]. You think about them. And you figure, is it good for me to do it at this particular time? or later? And the result is I am always going to do what I think is best for me. Regardless of what the doctor says. Because he doesn’t have my body. He assumes I have this type of pain, but does he know what I am feeling? … So you choose according to what you figure is best for you at any given time.

Participants described conversations they have had with their providers to make difficult treatment decisions, individualized to their own needs and other health conditions:

But I just could not function. So we changed. [My doctor] said, ok, we’ll work it another way…but I said no more of them, because the lungs was the hardest part, once they found out I had COPD and that was why I was going around sounding like a horse all day long. And to treat [my breathing] was more important than the little [benefit] I got [from the beta blocker for treating my] large heart with fatty tissue growing around it….[T]he beta blocker just made me feel worse.

Prevention of Adverse Long-Term Health Outcomes

Focus group participants considered the balance between potential short-term benefits, such as pain control, and the longer-term benefits, such as the prevention of kidney disease and heart attacks (“I don’t want the effects—the horrible effects—that diabetes brings. ‘Cause it brings them if you don’t watch it.”) versus the risks of side effects and treatment burden. One participant described her experience with arthritis, use of anti-inflammatory medications and her need for surgery, which had not completely resolved her arthritic pain:

[T]hey gave me an anti-inflammatory. But that anti-inflammatory… created some other problems. Subsequently, knee replacement was the answer to quiet the arthritis, but the arthritis…is still in the body and is not going to go away.


Through a six-step process using multiple stakeholder engagement, including patients and guideline developers, we identified two important treatment questions and the related patient-important outcomes relevant for people with MCCs, in order to inform the process of developing recommendations for CPGs. Patients with MCCs summarized overarching themes related to their care that should be addressed in CPGs. These themes included their desire to understand and discuss with providers the potential benefits and harms of treatments, wanting individualized treatments that addressed their unique health and life situation and valued function, energy and quality of life. Although our process was focused on choosing topics for evidence syntheses that can ultimately inform CPGs relevant for people with MCCs, we believe our process has the potential to inform other guideline development organizations on a strategy to include and engage patient stakeholders early in the process of guideline development.

Our findings show that people with MCCs desire patient-centered care that prolongs their lives, but also—most importantly—addresses the benefits and risks of treating all of their conditions as a part of the whole person to maximize quality of life. To summarize, one of our patient co-investigators stated that “I need doctors who are going to be able to treat me as a whole person, not look at me as different parts of this same body.” Despite the specific selection of MCC patients/caregivers with diabetes and hypertension, the list of outcomes that they identified were broadly applicable to multiple chronic health conditions. These outcomes are consistent with prior studies assessing the effect of treatment burden on people with MCCs.20 22 For example, Ridgeway and colleagues interviewed and conducted focus groups with people with MCCs, and these study participants also described the importance of emotional health and maintaining a positive attitude, and effective communication with providers about the risks and benefits of treatment.20 Notably, the majority of outcomes that were identified by the patients and caregiver focus groups are not currently addressed in CPGs. A strength of our process was that CPG developers had the opportunity to hear and reflect on the perspectives of patients and caregivers at each stage of the process, thus aiding in translating the patient voice to future CPGs.

Next Steps to Inform CPGs Related to MCCs

As our team begins the next step of reviewing evidence related to diabetes and hypertension in evidence syntheses, we will incorporate the outcomes identified in benefit/harm analyses to be subsequently addressed in CPGs. In addition, there could be a downstream effect on future guidelines not related to these topics. As CPG developers listen to and reflect the patient perspective in their guidelines, they will identify gaps in the literature around the effectiveness of treatments for many conditions in people with MCCs. CPG developers will be able to include and address patient-important outcomes, which could lead to additional proposals, funding and evidence generation to close these evidence gaps.

We identified several limitations of our study. First, qualitative research is not generalizable, but rather aims to generate hypotheses. The focus group participants were all patients of Kaiser Permanente Colorado, and their perspectives may differ from those of patients with MCCs in other health care delivery settings. In addition, individuals who participate in focus groups may be different from and potentially more activated and engaged in care than those who choose not to participate or do not have access to do so. Second, our work was limited to the Kaiser Permanente National Guideline Program, and other guideline-producing professional or health care organizations may have different processes. Kaiser Permanente’s National Guideline Program uses a process consistent with the standards recommended by the IOM in its high-quality systematic reviews and clinical practice guidelines. However, given the IOM reports, it is important for all guideline developers to consider how to integrate patients and other stakeholders into their recommendation process. Third, we started with a list of high-priority topics that had been previously generated by the ICVH guideline program without patient stakeholder input, and we limited the study a priori to select two high-priority topics to inform the upcoming quantitative analysis. Although the selected ICVH topics and outcomes resonated with our focus group participants and patient stakeholder co-investigators, our final list is not meant to be all-inclusive, as other health conditions, topics and outcomes may also be important to address. Conducting this project in partnership with a high-quality CPG program with rigorous methods was viewed as essential for these results to be both practical and applicable for other CPG developers, but necessitated working within their existing CPG process.

Our results have implications for systematic reviewers, CPG developers, researchers and policymakers working to improve patient-centered care for the large population of people living with MCCs. Our process provides a model for other CPG-developing organizations to meet the standards established by the 2010 IOM report, Clinical Practice Guidelines We Can Trust,12 and the Guidelines International Network.23 Several studies have proposed methods for eliciting patient preferences for care and for engaging stakeholders in the research process.24 26 However, we are not aware of studies that have connected patient stakeholders with CPG stakeholders in the same process to hear each other’s perspectives to directly inform the next step of evidence syntheses. As part of this process, we encountered several challenges with the stakeholder engagement process, including adding patient and caregiver stakeholders to the institutional review board protocol, involving them in large conference calls (vs. more personal meetings) and communicating the technical research methods. These challenges were addressed by facilitating the necessary human subjects protection training by the study team for patient and caregiver stakeholders, having a combination of in-person meetings with study co-investigators and virtual meetings with the entire team, and focusing on describing research methods at each patient and caregiver co-investigator meeting. The value of patient and caregiver participation in the process was solicited from the larger study team and communicated back to the patient and caregiver team members.

In conclusion, we have described a feasible, potentially replicable mixed-methods process involving multiple stakeholders for identifying two high-priority questions and relevant clinical outcomes for people with MCCs. These questions related to diabetes and hypertension will be the focus of future CPGs that address people with MCCs and incorporate the benefit/harm balance into the outcomes identified by patients and caregivers.



Research reported in this article was funded through a Patient-Centered Outcomes Research Institute(PCORI) Award (ME-1310-07619). The statements presented in this article are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.


  1. 1.
    Anderson G, Horvath J. The growing burden of chronic disease in America. Public Health Rep. 2004;219(3):263–70.Google Scholar
  2. 2.
    Goodman RA, Posner SF, Huang ES, Parekh AK, Koh HK. Defining and measuring chronic conditions: imperatives for research, policy, program, and practice. Prev Chronic Dis. 2013;10:E66.PubMedPubMedCentralGoogle Scholar
  3. 3.
    Boyd CM, Darer J, Boult C, Fried LP, Boult L, Wu AW. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. JAMA. 2005;294:716–24.CrossRefPubMedGoogle Scholar
  4. 4.
    Lugtenberg M, Burgers JS, Clancy C, Westert GP, Schneider EC. PMC3197602; current guidelines have limited applicability to patients with comorbid conditions: a systematic analysis of evidence-based guidelines. PLoS One. 2011;6:e25987.CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Mutasingwa DR, Ge H, Upshur RE. PMC3135464; how applicable are clinical practice guidelines to elderly patients with comorbidities? Can Fam Physician. 2011;57:e253–62.PubMedPubMedCentralGoogle Scholar
  6. 6.
    Tinetti ME, Fried TR, Boyd CM. Designing health care for the most common chronic condition--multimorbidity. JAMA. 2012;307(23):2493–4.CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Tinetti ME, Bogardus ST Jr, Agostini JV. Potential pitfalls of disease-specific guidelines for patients with multiple conditions. N Engl J Med. 2004;351(27):2870–4.CrossRefPubMedGoogle Scholar
  8. 8.
    PCORI National priorities and research agenda. Available at: Accessed February 23, 2017.
  9. 9.
    Uhlig K, Leff B, Kent D, et al. A framework for crafting clinical practice guidelines that are relevant to the care and management of people with multimorbidity. J Gen Intern Med. 2014;29(4):670–9.CrossRefPubMedPubMedCentralGoogle Scholar
  10. 10.
    Boyd CM, Vollenweider D, Puhan MA. Informing evidence-based decision-making for patients with comorbidity: availability of necessary information in clinical trials for chronic diseases. PLoS One. 2012;7(8):e41601.CrossRefPubMedPubMedCentralGoogle Scholar
  11. 11.
    Qaseem A, Barry MJ, Denberg TD, Owens DK, Shekelle P, Clinical Guidelines Committee of the American College of Physicians. Screening for prostate cancer: a guidance statement from the clinical guidelines committee of the American college of physicians. Ann Intern Med. 2013;158(10):761–9.CrossRefPubMedGoogle Scholar
  12. 12.
    IOM (Institute of Medicine). Clinical practice guidelines we can trust. Washington, DC: The National Academies Press; 2011.Google Scholar
  13. 13.
    Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD. Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database Syst Rev. 2006;(3)(3):CD004563.Google Scholar
  14. 14.
    Fleurence R, Selby JV, Odom-Walker K, Hunt G, Meltzer D, Slutsky JR, Yancy C. How the patient-centered outcomes research institute is engaging patients and others in shaping its research agenda. Health Aff (Millwood). 2013;32(2):393–400.CrossRefGoogle Scholar
  15. 15.
    Cukor D, Cohen LM, Cope EL, et al. Patient and other stakeholder engagement in patient-centered outcomes research institute funded studies of patients with kidney diseases. Clin J Am Soc Nephrol. 2016;11(9):1703–12.Google Scholar
  16. 16.
    Piette JD, Kerr EA. The impact of comorbid chronic conditions on diabetes care. Diabetes Care. 2006;29(3):725–31.CrossRefPubMedGoogle Scholar
  17. 17.
    Delbecq A, Van de Ven A, Gustafson D. Group techniques for program planning: a guide to nominal group and Delphi processes. In: Linstone HATM, ed. The Delphi method: techniques and applications. Reading, MA: Addison-Wesley Educational Publishers Inc.; 1975.Google Scholar
  18. 18.
    Quan H, Sundararajan V, Halfon P, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43(11):1130–9.CrossRefPubMedGoogle Scholar
  19. 19.
    Crabtree BF, Miller WL. Doing qualitative research. Newbury Park, CA: Sage Publications; 1992:276.Google Scholar
  20. 20.
    Ridgeway JL, Egginton JS, Tiedje K, et al. Factors that lessen the burden of treatment in complex patients with chronic conditions: a qualitative study. Patient Prefer Adherence. 2014;8:339–51.CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
    Bayliss EA, Balasubramianian BA, Gill JM, Stange KC. Perspectives in primary care: implementing patient-centered care coordination for individuals with multiple chronic medical conditions. Ann Fam Med. 2014;12(6):500–3.CrossRefPubMedPubMedCentralGoogle Scholar
  22. 22.
    Bayliss EA, Bonds DE, Boyd CM, et al. Understanding the context of health for persons with multiple chronic conditions: moving from what is the matter to what matters. Ann Fam Med. 2014;12(3):260–9.CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    Qaseem A, Forland F, Macbeth F, et al. Guidelines international network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156(7):525–31.CrossRefPubMedGoogle Scholar
  24. 24.
    Ryan M, Scott DA, Reeves C, Bate A, van Teijlingen ER, Russell EM, Napper M, Robb CM. Eliciting public preferences for healthcare: a systematic review of techniques. Health Technol Assess. 2001;5(5):1–186.CrossRefPubMedGoogle Scholar
  25. 25.
    Deverka PA, Lavallee DC, Desai PJ, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J Comp Eff Res. 2012;1(2):181–94.CrossRefPubMedPubMedCentralGoogle Scholar
  26. 26.
    Concannon TW, Meissner P, Grunbaum JA, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. 2012;27(8):985–91.CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Society of General Internal Medicine 2017

Authors and Affiliations

  • Wendy L. Bennett
    • 1
    • 2
  • Craig W. Robbins
    • 3
    • 4
  • Elizabeth A. Bayliss
    • 4
    • 5
  • Renee Wilson
    • 6
  • Heather Tabano
    • 4
  • Richard A. Mularski
    • 3
  • Wiley V. Chan
    • 3
  • Milo Puhan
    • 7
  • Tsung Yu
    • 8
  • Bruce Leff
    • 9
  • Tianjing Li
    • 2
  • Kay Dickersin
    • 2
  • Carol Glover
    • 8
  • Katie Maslow
    • 10
    • 11
  • Karen Armacost
    • 10
  • Suzanne Mintz
    • 10
  • Cynthia M. Boyd
    • 9
  1. 1.Division of General Internal Medicine, Department of MedicineThe Johns Hopkins University School of MedicineBaltimoreUSA
  2. 2.Department of EpidemiologyJohns Hopkins Bloomberg School of Public HealthBaltimoreUSA
  3. 3.Kaiser Permanente National Guideline ProgramDenverUSA
  4. 4.Kaiser Permanente, Institute for Health ResearchDenverUSA
  5. 5.Department of Family MedicineUniversity of Colorado School of MedicineAuroraUSA
  6. 6.Department of Health Policy and ManagementThe Johns Hopkins University School of Public HealthBaltimoreUSA
  7. 7.Epidemiology, Biostatistics and Prevention InstituteUniversity of ZurichZurichSwitzerland
  8. 8.Department of Public HealthChina Medical University TaichungTaichungTaiwan
  9. 9.Division of Geriatric Medicine and Gerontology, Department of MedicineThe Johns Hopkins University School of MedicineBaltimoreUSA
  10. 10.Informing Patient-Centered Care for People with Multiple Chronic ConditionsBaltimoreUSA
  11. 11.Gerontological Society of AmericaWashingtonUSA

Personalised recommendations