Consistent with typical GCM procedures, study procedures included the following phases: (1) preparation, (2) statement generation, (3) structuring, (4) representation, and (5) interpretation. As GCM is a linear process, the preparation, statement generation, and structuring stages are presented in the Methods section of this manuscript, while the final results of the GCM analyses (representation and interpretation) are presented in the Results section. This research was approved by the New England Independent Review Board (NEIRB).
Phase I: Preparation
The main goal of this phase is to refine the study focus, define the focus prompts, and to identify relevant stakeholders .
During an early phase of the RWJF grant, interviews with 26 individuals representing various interests in health care, including patient advocates (n = 7), purchasers (n = 6), clinicians (n = 4), researchers (n = 4), measure developers (n = 3), and health IT (n = 2) were conducted. The purpose of the interviews was to evaluate gaps in existing performance measurement efforts in PROs. Results from these interviews highlighted the need for a patient-driven conceptual model of health care. Information from the interviews was also used to develop the focus prompts and GCM statements (see below).
Focus Prompt Development
Based on insights gleaned from the stakeholder interviews, three open-ended prompts were developed to present in survey format to generate statements about good health care. Specifically: (1) Please give us statements that describe ‘good health care’ (2) What does ‘good health care’ look like? and (3) Think about the health care that you have received. What aspects of your care have you liked? Complementary survey items (e.g., How do patients describe ‘good health care’?) were developed for health stakeholders. Three prompts were utilized to minimize the possibility that item wording would bias responses, and prompts were presented in different orders to participants to compensate for order effects.
Phase II: Statement Generation
Statements describing good health care were generated from (1) PLM members, (2) health stakeholders, and (3) literature review. This method of triangulation was employed to ensure that the statement bank comprehensively captured all aspects of good health care.
Patients were recruited through PLM, an online research network that allows patients to share personal health information through structured data collection. PLM currently has over 600,000 members, representing 2900 different medical conditions.
PLM members were eligible for participation if they were 18 years of age or older, reported a primary residence in the United States (US), and reported having one of the following six health conditions: heart disease, cancer, stroke, diabetes mellitus, hypertension, and arthritis. It was anticipated that obtaining input from these diverse patient groups would increase the heterogeneity of responses and would facilitate generation of a comprehensive pool of statements that accurately reflect the construct of ‘good health care’.
Survey invitations were sent via electronic message to potentially eligible PLM members. The survey included questions pertaining to patient demographic/clinical characteristics (see Table 1), as well as the three open-ended good health care prompts. Members were not remunerated for participation.
After invites were sent, the research team continually reviewed survey responses to determine when saturation had been reached. PLM member statement generation was completed in five days in July 2016. Of the 994 PLM members who were invited to participate, 187 (18.8% response rate) responded to the survey and 157 provided usable data and were retained for further analyses. PLM members generated 1277 statements about good health care (Table 1).
Health stakeholders, identified through investigator and PLM contacts in health care institutions and companies through chain referral, were sent an email invitation to participate in the anonymous, voluntary survey. Health stakeholders were not reimbursed for their time.
Seventeen health stakeholders participated in statement generation. Of the participants, six represented patient groups, six were providers, one was a researcher, two represented purchaser groups, and two worked in measure development. These stakeholders generated 287 statements pertaining to good health care.
A literature review was conducted to identify previous studies that have evaluated patient health care priorities and to extract this content. Specifically, a search of the MEDLINE and PsychINFO databases of peer-reviewed articles since 2000 was conducted using combinations of the following keywords: ‘patient reported outcome,’ ‘performance measure,’ ‘quality of care,’ ‘quality of health care,’ and ‘health care quality indicators.’ Potentially relevant articles were reviewed by research staff and information pertaining to patient health care priorities or aspects of health care that are important to patients was extracted and compiled into a list of statements. References from relevant articles were also reviewed to increase comprehensiveness of the literature search. In total, 146 statements pertaining to good health care were generated from this literature review.
Secondary Review of Non-Patient Stakeholder Interviews
Statements describing good health care were extracted through secondary content analysis of health stakeholder interview transcripts that were conducted during the Preparation stage. From the 26 interviews, 69 statements pertaining to good health care were extracted.
Statement Pool Cleaning and Reduction
PLM member and health stakeholder responses to survey prompts were reviewed by the research team (authors of this study); compound responses where participants identified multiple aspects of good health care in the same response were separated for purposes of analysis.
To eliminate duplicate content and reduce the statement bank for GCM, statements were coded and grouped by content. The purpose of coding and grouping the statements was to more easily identify statements that were potentially duplicative. Two coders from the research team independently reviewed all statements and assigned each statement one or more keywords, iteratively developing a shared bank of keywords. Across the pool of statements, approximately 250 codes were iteratively and collaboratively generated by the raters. Raters achieved high levels (91%) of agreement. Disagreements were reviewed and discussed with a third researcher on the research team, who helped the team determine final codes.
Once final codes were assigned, statements were sorted by code to facilitate removal of duplicative content and to generate ‘prototype’ statements (i.e., the statement that most clearly reflected the primary content of that group of statements).
Whenever possible, language from the original statements was retained, although revisions were made in favor of clarity, grammar, and spelling. Additionally, six researchers provided qualitative and quantitative feedback using a grading system to facilitate reduction of the statement pool. Specifically, each statement was graded as A (very important for inclusion), B, or C (less important for inclusion) to efficiently identify the statements that were most important for inclusion in the final statement pool. The researchers had backgrounds in qualitative research, psychometrics, performance measurement, medicine, and health policy and included four PatientsLikeMe researchers and two consultants. This iterative and collaborative process resulted in a final pool of 79 statements.
Phase III: Structuring (Rating and Sorting)
CSGlobal MAX (©2017, Concept Systems, Inc.) software for GCM was utilized in the current study. The participant information, rating, and sorting modules in Concept Systems were utilized for the purposes of the current study.
Participants for Phase III include PLM members, Baltimore community members, and health stakeholders. Each group of participants is described separately below.
According to Trochim , meaningful results in concept mapping can be accomplished with as few as 10–20 participants per group. Therefore, waves of invitations (n = 500) were sent to PLM members who met the same inclusion criteria as the Statement Generation phase with the intention of obtaining complete data from around 120 participants (approximately 20 participants for each of the six disease groups). In total, 3266 members were invited and 193 PLM members provided complete rating and/or sorting data; 172 of these members completed the rating exercise and 123 completed the sorting. Following consent, PLM members were given access to the three GCM activities: (1) participant information, (2) rating, and (3) sorting. Patients were not reimbursed for their time. Demographic information from PLM participants who chose to provide it is presented in Table 2.
To diversify the Structuring sample and to ensure that patients who were not members of PLM or did not have access to a computer or the Internet were not excluded from this research, a collaboration was established with UMD’s PATIENTS program. The PATIENTS program, funded through the Agency for Health care Research and Quality (Grant # 5R24HS022135), is designed to empower patients to get involved in health care research. This program engages people from the surrounding Baltimore communities, especially underserved and minority populations. Community affiliates of the PATIENTS program who served as volunteers and community partners were recruited at community events and identified through other PATIENTS program staff. PATIENTS program staff were present during study administration to create de-identified accounts for participants and assist with data collection. Patients were reimbursed for their time with $25 gift cards. Patients were eligible to participate if they were 18 years of age or older and were currently residing in the US. From two rounds of recruitment and testing, 28 Baltimore community members provided usable data; 27 of these patients completed the rating and 27 completed the sorting activities. Demographic information for those participants who chose to provide it is presented in Table 2.
Health stakeholders in the field of performance measurement were invited to participate in structuring, including health stakeholder participants from previous phases. They were sent an email inviting them to participate with a link to the Concept Systems program to complete the sorting and rating activities, described above. Sixteen health stakeholders completed the rating and 15 completed the sorting exercises. The sample consisted of four members of purchaser groups (e.g., health insurance companies), five members of patient advocacy groups, three members of providers such as physicians, health psychologists, and researchers, and four measure developers.
The final statements were randomized and put into the Concept Systems program for purposes of ratingFootnote 1 and sorting. During the rating activity, patientsFootnote 2 and providersFootnote 3 were asked to rate the 79 statements on a scale of 1 = Not Important to 5 = Extremely Important. The sorting exercise required participants to read the list of 79 statements and to sort them into meaningful groups. Participants were asked not to sort statements according to priority or value or into a ‘miscellaneous’ or ‘other’ pile of dissimilar statements. All participants were instructed to complete the rating exercise prior to sorting, as completing the rating exercise (which required participants to read and consider the content of all 79 statements) could facilitate completion of the sorting task. However, participants were able to complete the exercises in the reverse order if they preferred.