FormalPara Key Points for Decision Makers

The Altarum Consumer Engagement (ACE) Measure™ is a 21-item measure that provides a valid alternative to other patient engagement measures.

The ACE Measure™ consists of four scales:

 Commitment: confidence and ability to maintain a healthy lifestyle and manage one’s health.

 Ownership: perceived role in and responsibility for one’s health.

 Informed Choice: patterns of seeking and using information about health and healthcare.

 Navigation: confidence and ability to ask about and participate in treatment decisions.

The scales are predictive of important outcomes, including patient health, self-reported medication adherence, and use of online decision tools.

1 Introduction

Consumers of healthcare are being asked to play a more significant role in their care than ever before, as seen with the advent of the patient-centered medical home model, the expansion of coordinated care for chronic disease, and the growth of models for shared decision making. Moving patients into this role is often discussed under the topic of engagement, a concept that is increasingly popular in the professional and scientific literatures. Despite the attention to the concept, there is a wide diversity of opinion on what engagement actually means, and what range of behaviors and attitudes it encompasses. Health information technology (IT) advocates, for example, might view a patient as engaged when they pick up laboratory results through a web-based patient portal [1]. Chronic disease management identifies the engaged patient as one who is competent in self-care [2]. For the purposes of the study described here, we follow Gruman et al. [3] and adopt a broad, behavioral definition of engagement as “actions individuals must take to obtain the greatest benefit from the health care services available to them”.

Evidence shows that positive benefits accrue when patients (healthcare consumers) are engaged by this definition. Patient involvement in care decisions results in safer [4], more effective [5], and less expensive [6] healthcare. Patients who participate in care decisions report higher satisfaction [4], faster recovery from illness [7], and better quality of life. Additionally, care plans that come from a shared decision making process result in improved medication adherence [8] and clinical outcomes [9, 10]. Engagement also has notable protective effects on patient safety. Studies find that patients who reported the highest levels of active participation in their care have half the rate of medical errors in the hospital as those with low participation [4].

Patient involvement can also contribute to lower overall healthcare costs due to avoided unnecessary surgeries [6], better adherence [8], prevented medical errors [4], and better condition management that accompany active participation [8–10]. By giving consideration to individual differences in preference about such things as quality of life and length of life, economists estimate substantial potential savings from choosing cost-effective options based on individual preferences rather than choosing one “most effective standard” approach across all patients [11].

The rapid pace of change in the healthcare system presents significant challenges to keeping patients engaged. The expansion of high-deductible health plans in the USA increasingly requires patients to weigh price and quality information as they make decisions about where and when to seek care. Few measurement tools exist to help identify patients’ competence in this domain. While there are several survey instruments that measure one or more aspects of patient involvement [12–22], most have a specific emphasis, such as competence [13], autonomy [12, 14], activation [16], information preferences [20], or decision making [17]. Others cover multiple domains but focus on managing existing disease [22]. Currently available scales do not adequately address dimensions of engagement that map to skills patients need to take advantage of new information sources and tools. There remains a need for science that identifies factors that encourage and speed adoption of patient involvement, and evaluates programs and tools designed to facilitate patient choice making. To fill this gap, we developed the Altarum Consumer Engagement (ACE) Measure™ with the purpose of measuring a broad conceptualization of patient engagement and health consumerism, covering multiple psychometric and behavioral domains, and incorporating the use of modern information sources, such as healthcare comparison tools and online resources.

2 Aims

The aims of this study were to:

  • Test the factor structure and response format of a set of items developed for assessing multiple constructs of consumer engagement;

  • Test the factors identified for unidimensionality;

  • Determine the validity of each factor in predicting particular patient engagement outcomes; and

  • Test whether factors could be summed to produce an overall score.

3 Methods

3.1 Development of the Altarum Consumer Engagement (ACE) Measure™ and Subscales

In developing the ACE Measure™, our goal was to cover multiple aspects of patient perceptions and activities, including not only participation in health and healthcare but also use of information to compare and choose providers or services.

The first phase of development was review of existing literature and scales followed by expert input. Based on review and input from industry experts, an emphasis was placed on self-management skills and health system navigation, for both experienced and inexperienced healthcare consumers. Further, specific focus was on patients’ ability to compare and select care providers. Next, 95 items were created to cover many aspects of patient behavior and perceptions within the above-mentioned areas. Through Principal Component Analysis (PCA) in an initial pilot study survey, and review of the emerging factors, this list was reduced to 29 items. To fortify the areas of system navigation and provider comparisons, five items were added.

This set of 34 items was administered via a web portal to a general population panel of US adult respondents in November 2013, managed by Survey Sampling International. To test predictive validity, respondents were asked about their general health, their weight, whether they had ever asked about the price of a medical service before receiving that service, and whether they sometimes forget to take their regular medication. In addition, to test criterion validity, respondents were asked to score 13 items that are contained in the Patient Activation Measure® (PAM)® [16].

3.2 Statistical Analysis

Respondents with more than 10 % missing values were removed from the analysis dataset, representing 41 cases [23]. PCA (Oblimin rotation) was used to identify suitable subscales. Initial eigenvalues >1 determined the optimum number of factors, and a threshold value of 0.4 was used for item loading coefficients in the analysis [24]. Scale reliability was assessed using Cronbach’s α coefficients [25].

Using appropriate software and methods [26], Rasch analysis [27, 28] has become increasingly accepted across all research disciplines as an improved method of developing scales [29–31]. Fitting data to the Rasch Model addresses several key methodological aspects associated with scale development and construct validation, as well as providing a linear transformation of the ordinal raw score [31]. In other words, it tests whether related constructs can be summarized in a meaningful composite score.

The data were analyzed in RUMM2030 software to test the unidimensionality of individual subscales using Rasch Analysis [26]. We used the Partial Credit Model, which allows items to have varying numbers of response categories and does not assume the distance between response thresholds is uniform. As the Chi-square (χ 2) statistic tends to be sensitive to large sample sizes and fit residuals can become unstable [32], a random sample of 500 survey respondents was taken from the source dataset for testing goodness of fit (GOF). To check for agreement, a series of follow-up analyses were conducted on sample sizes of 250, 500, 750, and 1000 respondents.

A well-fitting solution would be indicated by a probability from the Item–Trait Interaction χ 2 < 0.05, after dividing by the number of items in the scale (Bonferonni correction) [33]. Individual item–fit residual (IFR) values greater than +2.5 were taken to indicate misfit and less than −2.5 to indicate item redundancy. Internal consistency was assessed by the use of the Person Separation Index (PSI), with values >0.7 considered acceptable for group-level analysis. Re-scoring was considered if it was thought there would be a significant improvement in model fit. Differential Item Functioning (DIF) was checked for possible item bias, caused by the responses by different groups in the sample. This study assessed DIF for respondent sex, age group, and PAM level. Person–item threshold maps were plotted to assess whether the ACE Measure™ subscales appropriately targeted the respondent group. Lastly, dimensionality was assessed by equating t tests to compare person estimates derived from the two most disparate subsets of scale items [34]. A threshold level <5 % was considered acceptable.

For each of the subscale scores, a subtest analysis was conducted to test whether the unidimensionality assumption would hold when the individual subscales were merged. Items were introduced to the analysis as subsets and assessed GOF using exactly the same methods as for the individual subscales.

4 Results

A total of 2079 records were used for analysis (Table 1). No restriction in range, floor, or ceiling effects were indicated for any responses at item level. Item descriptive statistics were checked for skewness and kurtosis and no evidence was found of outliers.

Table 1 Participant demographics and frequencies
Full size table

4.1 Principal Component Analysis and Rasch Analysis

A pool of 34 items were analyzed with PCA using Oblimin rotation. After deleting items that did not load on any factor >0.4, an initial 5-factor solution emerged, each containing between four and seven items. This solution was then examined by Rasch Analysis. One factor (composed of four items about decision making) showed a highly significant deviation from the Rasch model (p < 0.0005) and, despite the stepwise removal of items and re-assessment, unidimensionality could not be demonstrated. No further analysis was conducted on this scale and the items were not carried forward to the final solution.

4.1.1 Factor 1: Commitment

PCA suggested an initial subscale composed of seven items. Inspection of the threshold map confirmed that there was no disordering of thresholds. Item redundancy was indicated for one item (“Can accomplish goals”, IFR of 4.162). Once removed, all GOF statistics improved to within accepted limits, with a good PSI of 0.81. Percentage of equating t tests significant was 3.00 %, supporting unidimensionality of the scale. No DIF for respondent sex or age could be found (Bonferonni correction, p > 0.004).

4.1.2 Factor 2: Informed Choice

The threshold map for the six-item subscale showed no disordering but initial GOF was poor and identified one item (“Worry about privacy”) with significant misfit (IFR = 6.99) for removal. All GOF statistics moved into the normal expected range, with a good PSI of 0.82. Equating t tests of 7 % (95 % CI 0.05 to >0.09) supported unidimensionality of the scale. DIF was found for two items according to sex of respondent. For the item “Read health magazines for information”, the female respondents were more likely to endorse this item at the lower end of the Informed Choice Scale than the males. For the item “When choosing a new doctor, I look for official ratings based on patient health”, the male respondents were more likely to endorse this item at the lower end of the Informed Choice Scale than the females.

DIF was also present for the items “Read health magazines to get information” and “Compare official ratings about how well patients are doing” across age groups. Younger respondents were less likely to endorse “Read health magazines” at the higher end of the Informed Choice Scale. The item “Compare official ratings about how well patients are doing” showed DIF for age, with younger participants more likely to endorse this item than older participants.

4.1.3 Factor 3: Navigation

The initial Factor 5 subscale included five items scored on a 5-point scale. Inspection of the threshold map confirmed that responses to one item (“Comfortable talking to my doctor”) showed slight disordering. The initial GOF χ 2 was good and all item and person fit statistics were within range. No changes in this scale were indicated, with a PSI of 0.72 (see Table 2). DIF was found for the item “Comfortable talking to my doctor” according to age (Bonferonni correction, p < 0.0033). For this item, the younger respondents were less likely to endorse this item at the lower end of the Navigation scale than the older respondents.

Table 2 Rasch Overall fit, item, and person fit statistics for the four subscales and overall Altarum Consumer Engagement (ACE) Measure™ total score
Full size table

4.1.4 Factor 4: Ownership

The original Factor 4 subscale included five items, with one item (“Own responsibility”) showing slight disordering in responses. The initial GOF χ 2 was significant at p = 0.002, all except one of the items and person fit statistics being within accepted limits. The PSI was 0.686 and percentage of t tests was 4.6 %, which supports the unidimensionality of the scale. No DIF for respondent sex or age could be found (Bonferonni correction, p > 0.004).

4.2 Subtest Analysis for Combining Factors Into a Single ACE Measure™ Score

Subtest analysis was conducted using 21 items which composed the four subscales above. Overall fit for the summary score was good (χ 2 = 27.3, df = 28, p = 0.503). All summary fit statistics showed an excellent GOF to the Rasch model; item fit [mean = 0.064, standard deviation (SD) = 0.725] and person fit (mean = −0.455, SD = 1.066) with a PSI for group analysis of 0.899. Equating t tests of 7 % (95 % CI 0.05 to >0.09) supports unidimensionality of the ACE Measure™ total score, indicating that the four ACE Measure™ domains can be aggregated into a single score.

4.3 ACE Measure™ Scoring Procedures

Each of the 21 questions (Table 3) was scored from 0 to 4. For each subscale, the mean score was multiplied by 6.25 to create a subscale score from 0 to 25. By summing the four subscale scores, a total ACE Measure™ score was calculated, ranging from 0 to 100. The overall mean for the ACE Measure™ total score was 66.9 (SD = 12.8). The median score for this group was 77 (range 27–105, Skewness = −0.032 and Kurtosis = 0.415).

Table 3 Altarum Consumer Engagement (ACE) Measure™ subscale items and scale reliability
Full size table

To test whether each subscale reflected a unique construct relating to engagement, we examined the variation in scores according to demographic and behavioral outcomes (Table 4). We hypothesized that the Commitment subscale reflects proactive self-care behaviors and consistency under typical and stressful circumstances. Informed Choice refers to activities and sources of information a person investigates to compare and choose among providers or services. The Navigation scale indicates the comfort level and experience the person has in asking for and giving input about the types of care they will receive within the healthcare system. Ownership refers to the degree to which the person assigns responsibility for health outcomes to their own actions.

Table 4 Validation: analysis of scales by demographic variables
Full size table

As expected, there were no differences in the scores on the ACE Measure™ scales by sex of respondent. The only difference in ACE Measure™ subscales among age groups was that respondents over 50 years old were less likely to report activities such as seeking information, making comparisons, and choosing among care providers.

All subscales and the ACE Measure™ total score were positively associated with income, perhaps because of greater education, access to resources, and other factors. As expected, those reporting better health status also had higher scores on every ACE Measure™ subscale. Because higher engagement could be expected to lead to improved health management skills and greater involvement in care decisions, this result was not surprising.

To assess predictive validity, we tested for relationships between the ACE Measure™ subscales and different health outcomes. Respondents who reported being ≥10 lb (4.5 kg) overweight scored lower on the overall ACE Measure™ score and all subscales except Navigation. Being overweight was most strongly associated with the Commitment scale, which reflects self-management skills. Because the Navigation score pertains to participation and experience in decisions within the healthcare system, it was reasonable to find that these skills were not related to lifestyle factors such as weight.

Those who reported asking about the cost of health services before treatment scored higher on the Commitment, Informed Choice, and Navigation scales (Table 4). This was consistent with the hypothesis that those subscales were related to participation in activities related to the care system. Respondents who reported forgetting to take their medicine scored significantly lower on the Commitment subscale, which was most reflective of self-management activities. Again, this supported this scale’s indication of the individual’s own self-care.

To assess criterion validity, we compared the ACE Measure™ subscales to the well-established PAM. All subscales differed significantly across levels of PAM (see Table 5). The Commitment subscale was most strongly associated with the PAM (Spearman’s Rank correlation coefficient, r s = 0.65). Informed Choice was least associated with PAM, with a correlation of r s = 0.29.

Table 5 Validation: tests of significance between respondents in the different PAM groups (levels 1 to 4)
Full size table

5 Discussion

Our purpose in this study was to develop a measure of patient engagement and healthcare consumerism that would encompass multiple domains of engagement for all types of patients. We wanted to measure more recent developments in engagement behaviors, including published healthcare ratings and use of comparison tools [3]. Existing measures of patient engagement [12–22] are unidimensional [16], highly focused on a single aspect of engagement [12–14, 17, 20], or meant for disease management rather than a broad population [22], and none of them measure usage of published healthcare ratings or comparison tools.

This study has confirmed the psychometric properties of the ACE Measure™ in a broad sample of US adults and illustrated its utility in screening for and finding group differences in consumer-oriented health activities. The 21-item instrument consists of four valid subscales, each indicative of a unique aspect of engagement. However, the combined summary score can also be used as a unidimensional indicator of overall engagement, as supported by the Rasch subtest analysis. This validation focused on a broad cross-section of US adults; further research will be needed to validate the ACE Measure™ in a variety of settings, such as in acute care, and those managing chronic illness.

We chose a Rasch analysis because of its advantages over more traditional forms of dimension reduction, such as factor analysis or principal components analysis alone. Rasch analysis is superior for analyzing ordinal data, which includes most Likert-style survey data. Additionally, Rasch analysis can determine how item scores may differ based on demographic variables, such as the effect of age on Informed Choice. Finally, Rasch analysis can test for both multidimensionality and unidimensionality within the same question set.

A patient engagement score, as measured by the ACE Measure™, is related to some aspects of demographics, but not all. The Rasch analysis showed DIF for sex and/or age on three of the 21 items, primarily relating to age and the Informed Choice factor. Older respondents were less likely to use ratings and comparison tools to identify providers or healthcare. Because of this, patient age should be taken into account when considering the Informed Choice scale, with the understanding that the older patients may be less likely to use ratings or comparison tools despite an otherwise high interest in shared decision making. However, because use of healthcare ratings are included in the first element of the Engagement Behavior Framework [3], we felt this was an essential component of patient engagement absent from other engagement measurement tools, and should remain in the scale despite the association with patient age.

None of the ACE Measure™ subscale means differed by sex of respondent. Three of the ACE Measure™ subscale means did not differ by age; only Informed Choice scores were significantly higher for younger respondents. This suggested that most attributes of engagement were determined independently from basic demographic characteristics. However, all scale scores in the ACE Measure™ were significantly higher for respondents with higher incomes. Because higher incomes are related to many known determinants of health, this could reflect education, resources, or other opportunities that could influence engagement. Similarly, all subscales of the ACE Measure™ were strongly related to self-reported health status. Those in good or excellent health scored significantly higher on all measure subscales than those in poor or fair health.

Each of the subscales had utility identifying specific aspects of patient engagement behaviors and attitudes. For example, higher scores on the Commitment subscale were associated with healthy behaviors, such as maintaining a healthy weight and adherence to a regular medication regimen. Conversely, medication adherence was not associated with Navigation or Informed Choice scores.

Scores on Informed Choice indicated a higher likelihood of comfort with using modern information sources to compare care. Navigation scores were not associated with healthy behaviors, but were reflective of actions related to care choices. Specifically, those who scored highly on Informed Choice and Navigation were more likely to have asked about the cost of medical care before treatment. The ACE Measure™ scales were significantly correlated with the PAM, a widely used instrument focused on “patient knowledge, skills and confidence for health management” [16]. As would be expected, the strongest association was between PAM and the Commitment subscale, which focuses on health management activities. Average respondents in Level 1 of the PAM categorization scored 11.8 on the commitment scale compared with 20.1 for those in Level 4.

The lowest correlation was observed between the PAM and Informed Choice scores, with a correlation coefficient of r s = 0.29, which indicated only a moderate association. One of the original goals of developing the ACE Measure™ was to create a metric associated with activities related to information-seeking and use of modern decision-making tools, such as online health resources and published ratings of providers and patient health, which has not been the focus of other measures. Given the low correlation, this suggested that this subscale was a novel construct compared to PAM.

This study aimed to create a new measure of patient engagement and healthcare consumer behaviors that would encompass multiple domains, including the use of healthcare ratings and comparison tools, which was absent from the current tools available. This study has established the psychometric and predictive qualities of the ACE Measure™ in relation to specific health outcomes and behaviors in a general survey of US adults. Future research should continue validation of the ACE Measure™ in a range of contexts, such as with understanding the needs of acute care patients, tailoring the management of chronic disease, or testing the effect of engagement intervention programs.

6 Conclusion

The ACE Measure™ offers an alternative tool for assessing patient engagement and consumerism behaviors in healthcare. It contains four subscales, each of which measures a unique aspect of engagement. The Commitment scale relates most strongly to traditional health management outcomes, while the Informed Choice scale indicates active seeking and use of information about care options not measured in existing tools. Rasch Subtest Analysis has confirmed that the four subscales can be aggregated into a meaningful total score indicating overall engagement. This study supports the use of the ACE Measure™ in predicting a range of health outcomes and behaviors in a general population of US adults, but additional study and validation is recommended, such as in acute care settings and with the management of chronic disease.