Background

Health related quality-of-life assessment methods are increasingly used to develop indices that can support health economic evaluation of stroke care. Utility (or preference-based) measures, typically reporting on a single 0 to 1 scale, based upon decision and utility theories, are designed to elicit the value people place on a particular health state. Such preference-based approaches integrate different aspects of health into a single index, usually anchored by a value of `1.00’ for full health and `0’ for death. These measures are increasingly important since they are used to actively inform both health policy and individual decision-making. At a population level, they support resource allocation decisions with their use in health technology assessment and economic evaluation that lead to the production of guidance to health services regarding the use of health interventions e.g. in England and Wales guidelines are produced by National Institute for Health and Care Excellence (NICE) [1]. Furthermore, these measures are adopted in guideline production in a number of different countries including Australia [2] Canada [3] and the United States [4]. They have also been used to support decision making at an individual level, e.g. by incorporating decision analytical models in clinical decision support systems and patient decision aids [5],[6].

Two approaches to utility elicitation exist - indirect and direct. In indirect elicitation patients complete a questionnaire, such as the EQ-5D, which is mapped onto utility scores previously developed. Indirect methods are less time consuming for respondents, but lack content coverage and are often insensitive to change [7]-[9]. Direct methods are more complex and time consuming, but it is suggested that they may be more reliable, valid and responsive [10]. They require health state descriptions for specific health states such as major or minor stroke (examples are shown in Table 1), and derive a value by taking respondents through a valuation exercise, such as visual analogue scale (VAS) standard gamble (SG) or time trade off (TTO) [11],[12]. These approaches differ in the way they are undertaken, but all seek to derive a value between 0 and 1 for the relevant health state by taking people through a guided exercise.

Table 1 Examples of health state descriptions
Full size table

Equally, it is increasingly recognised that health services and policy need to reflect the patient and public perspective through patient centred and personalised care [15]. This implies that any such values derived from patients and the public, and used to support decision-making, should be valid and reliable, and appropriately reflect lived patient experience.

It is widely reported that direct utility estimates vary considerably, which might reflect the method used or the way the health states are described [16]. Specifically for stroke, variations in estimates resulting from direct preference elicitation have previously been explained by a number of factors. First, the choice of study population; for example, healthy participants assign lower utilities than patients who have experienced stroke [11],[17]. Second, the method of elicitation, with standard gamble derived scores being generally higher than those derived from time trade off studies, which in turn are higher than those derived from visual analogue scales [16],[18]. Third, the bounds of the scale, typically 0-1, may be defined differently: the upper bound defined as perfect health yields lower utility scores than if defined as the absence of the condition of interest, which is no guarantee that health is perfect [19].

Variations in health state descriptions content may also explain differences. The level of detail in health state descriptions can affect utility scores; longer, more comprehensive, descriptions (although appearing to have more face validity), can overload respondents’ cognitive capacity so that they latch onto a few key phrases and ignore the rest [20]. Naming or labelling a condition may have an impact [20]-[23] with a recent study recommending avoiding condition labels in health state descriptions to ensure that values are not affected by prior knowledge or preconceptions of the condition that may distort the health state being valued [10]. Furthermore, the wording may also cause variation in values if not presented in a balanced manner (framing bias), with both positive and negative effects described; explicit inclusion of negative aspects of stroke elicit lower values [24],[25].

Despite this, little attention has been given to the appropriateness of health state descriptions. If they don’t adequately reflect the reality of the health states they seek to measure, decisions based on these derived estimates may be compromised. Hence, at the very least, descriptions should reflect the reality of living with a stroke. Therefore, we set out to determine whether published stroke health state descriptions used in value elicitation studies truly reflect patients’ experiences of stroke by comparing their content with the results of qualitative studies exploring patients’ lived experience of stroke.

Methods

Two literature reviews were conducted: stroke health state descriptions used in direct preference elicitation studies and the qualitative literature on patients’ experience of stroke. The former review sought to comprehensively capture published health state descriptions for stroke used in preference elicitation studies; the latter, to capture what is important to patients in their lived experience of stroke from qualitative studies of patient perspectives. Both reviews involved the use of search strategies that included a combination of both subject headings and relevant key words.

Search strategy for health state descriptions

MEDLINE, CINAHL and PsychInfo were searched (1980 to 2011) using the search terms: stroke, cerebrovascular accident, cerebral arterial diseases, cerebrovascular disorders, cerebral thrombosis, carotid artery thrombosis, cerebral haemorrhage, cerebral hematoma, apoplexy, hemiplegia and hemiparesis. These terms were combined with the following terms: utility, quality of life, preference elicitation, time trade off (TTO), standard gamble (SG), Quality Adjusted Life years (QALY), stroke preferences, cost-effectiveness analysis and cost-utility analysis. Reference lists of all included articles were also searched.

Articles were included if they used appropriate direct preference elicitation methods, were in English and included the wording of the health state description. One reviewer performed initial selection (JG). Two reviewers (ML and RT) independently assessed studies for inclusion and extracted data, with disagreement resolved by discussion.

Search strategy for patients’ experience of stroke

MEDLINE, Embase, ISI Web of Knowledge and PsychINFO were searched (1997 to 2011) using the search terms: stroke, experience and qualitative, supplemented by hand searching reference lists from papers identified in both searches and related PubMed citations. Studies were included if: the findings focused on experiences of patients with stroke; were in English; used accepted qualitative methods; provided a clear exposition of methods and data collection; were supported by direct quotations; and were deemed of sufficient quality. Quality of the studies was based on criteria established by Mays and Pope [26]. Articles were scrutinised by two experienced qualitative researchers (MM, ML) and were included in the review if they were based on the appropriate application of established qualitative methods -that data collection, analysis and sample selection were appropriate to the explicit or implicit research question(s) - and contributed to knowledge in the field. Studies that did not provide sufficient detail of methods or those reporting opinion, but not providing direct empirical evidence, were rejected. Articles on the experiences of carers and professionals, trial participation, and assessments of rehabilitation therapies, information provision, and healthcare and community services were excluded. Articles focusing on specific characteristics of stroke experience such as end of life, pain, communication, return to employment or wheelchair use were excluded to avoid the data being skewed to one particular aspect of the stroke experience. Results and discussion sections were extracted for use in content and thematic analysis. The extracted data were coded as described below.

Content analysis and interpretation

Content analysis [27]-[29] was used by MLSL to identify the characteristics of stroke experience included in health state descriptions and in the results/discussion sections of qualitative studies. Using an inductive approach, text describing the patient experience was subjected to open-coding and category creation with the help of NVIVO software [30]. A comparative analysis was conducted to ensure the distinctions between the categories and the consistency of the content coded within each of the categories as well as across the two sets of data sources i.e. preference elicitation studies and qualitative literature. Qualitative inter-rater checking of definitions of the categories and the coded text was carried out by JG and discussed by the research team. Data regarding counts of the number of studies that included each category was extracted in order to undertake a comparative analysis of both types of study in terms of these categorised counts. The categories and counts of study were further mapped onto four predefined domains of HRQOL that encompass relevant areas from the stroke patients’ perspective as being crucial to quality of life measurement [7],[30]: biophysical, mood and cognition, prognosis and social domains.

In order to interpret the experience of stroke represented by the characteristics defined in the content analysis we also undertook a thematic synthesis of the qualitative studies [31]. This method includes systematic searching of the literature, quality assessment (as described above), extraction of data and thematic analysis of those data, i.e. familiarisation and coding line by line to develop descriptive themes. The text of results sections of each qualitative article formed the data for analysis. The thematic analysis conducted by MJM was an inductive process which followed the six stages described by Baun and Clarke [32]: familiarisation, generation of initial codes, searching for themes, reviewing themes, defining and naming themes and reporting the analysis. Here we term these `categories’ for ease of comparison with the content analysis. Content and thematic analyses of categories are integrated in the results below. Interpretation of these themes forms the analytic component of the analysis and is reported in the discussion.

Results

Health state descriptions

Seven studies that used direct preference elicitations were included (Table 2) [13],[14],[17],[24],[33]-[35]. Research participants included stroke survivors, those at increased risk of stroke and healthy people. Health descriptions ranged from mild/minor, moderate to severe/major stroke, examples of which are presented in Table 1, with standard gamble and time trade off the most frequent methods.

Table 2 Summary of preference elicitation studies
Full size table

Patients’ experience of stroke

Twenty eight qualitative studies examining post stroke experience were included [37]-[64] (Table 3) and were conducted in the UK (n = 9), the USA (n = 7), Norway (n = 4), Sweden (n = 4), Canada (n = 3) and Australia (n = 1). Methods predominantly comprised one-to-one interviews, with two studies employing focus groups [37],[38]. The theoretical perspectives for analysis included phenomenology [39]-[46], grounded theory [37],[39],[47]-[49], narrative analysis [50]-[52] and discourse analysis [53]. Participants were predominantly over 60 and Caucasian, although US studies included Hispanic and African Americans. Two studies [49],[51] included a small number of Bangladeshis and African Caribbeans. Specific sub-populations were targeted in seven studies: elderly non-institutionalised women [54], women in a rural setting [41], patients in the community [55], working class men [43], young women stroke survivors [56] and US war veterans [47],[52].

Table 3 Summary of qualitative studies
Full size table

Content analysis

Fifteen coded categories were identified in the preference elicitation studies (Table 4). The categories included varied across the studies, with only paralysis and dependence included in all. Only three studies made reference to continuing or worsening disability [13],[14],[17]. Toileting [14],[35], care arrangements [24],[14] and mortality [17],[24] were identified in only two studies. Solomon et al. [24] included the most categories within their descriptions and this was the only study to include pain and receptive problems. The following is an example of coded text under the category “Receptive problems”:

Table 4 Categories included in health state descriptions
Full size table

“You suffer a stroke that takes away your ability to understand language. You no longer understand anything being said to you” [24]

The sparest thematic content was found in Samsa et al. [17], although this only included major stroke, described as:

“a stroke that leaves an arm, a leg, and one side of your body paralyzed, and leaves you unable to take care for yourself. Anyone who has a major stroke will stay in this state until death”.

In addition and in order to help assess the relative impact of aphasia on preferences associated with major stroke, approximately 50% of the interviewees were randomly assigned to include the inability to speak in the description of the sequelae of a major stroke.

Information sources used to develop health state descriptions varied, but there was no reference to stroke patients’ perspectives; no studies included primary research with patients to ascertain them. Three studies [25],[26],[65] used an existing functional outcome scale - the Modified Rankin Scale (mRS) - which ranks levels of disability [50] to inform descriptions, with only one [25] citing further evidence [51] to support domains of function included. The scaling validity of the severity classifications for one study [30] were tested by neurologists specialising in stroke care. Two studies [28],[29] made no reference to how the descriptions were developed. One study [27] suggested that the descriptions were adapted from a previous study [25].

Differences in content across studies could not be explained by variations in stroke severity. Of the three studies that utilised the mRS to inform the descriptions for levels of stroke severity [14],[33],[35]], two [33],[35] provided descriptors for mild, moderate and severe/major stroke with categories common to both including: paralysis, dependence, feeling weakness, numbness or tingling, mobility and ambulation, expressive problems, memory and thinking, and facial droop. However, coordination and dexterity, and returning to normal activities, were only included in one study [33], and toileting only included in one other study [35]. Furthermore, mild/minor stroke was defined by different levels of mRS across studies (mRS = 1 [35], mRS =1-2 [33] and mRS =2-3 [14]).

Twenty nine coded categories were identified in the qualitative literature (Table 5). Counts of studies including each category showed that change in self identity and social role was the most frequently cited category (n = 26, 93%), followed by emotional difficulties (n = 25, 89%), mobility and ambulation (n = 24, 86%), and returning to normal regular activities (n = 24, 86%). The following are two examples of data from qualitative literature coded under `Change in self-identity, social role’:

TBW “Are you thinking about the fact that you had a stroke when you say you have changed, or are you thinking more in general?”

R “No, since I had the stroke. I don’t recognize myself. It is awful. You are in a way degraded. I am, even though you cannot see anything on me. Everybody says that I’m so not and so on. There’s no help in that. Nobody realizes how I am in reality.” (Case 4) [58]

Table 5 Rank ordering of categories by counts of study and study type
Full size table

In another example loss of physical function leads a patient to struggle with his sense of who he is in conjunction with his prestroke identity, as he recounts here:

“The one thing that’s very difficult for me as a person. . . I cannot relate, or quickly relate, back to where I was before I had the stroke. So, that comparison, I just can’t get it through my head to let that go, that I can’t do that.” (Mr. H. N.) [55]

Comparison of health state descriptions and patients’ experience: thematic synthesis

Mapping categories onto domains related to HRQOL for both study types resulted in four domains and associated thematic content (Table 6): biophysical (including 11 categories), mood and cognition (six categories), prognosis (four categories), and social (eight categories).

Table 6 Domains and categories by counts of study and study type
Full size table

Qualitative studies included a wider range of categories in every domain than health state descriptions. Health state descriptions missed categories in every domain that were relevant to patients’ experience, although all categories included in the health state descriptions were identified as important to patients in the qualitative studies . In each domain, the most often cited category differed between health state descriptions and qualitative studies and, with the exception of the biophysical domain, the most often cited categories in the qualitative studies (emotional difficulties; recovery, getting better; change in self-identity) did not appear at all in the health state descriptions.

A key feature of patients’ experience of stroke was the unanticipated, and therefore disruptive and sometimes shocking, nature of the experience. This carried through into the recovery phase, not only in the form of uncertainty about long term survival, but also in the potential for the disruption of everyday life. Categories in the biophysical domain reflect this potential disruption. Within this domain, paralysis was cited most often in the descriptions, in contrast to mobility and ambulation in the qualitative studies. Furthermore, dizziness and fainting, effects on sight, loss of swallow and fatigue, that were prominent in the qualitative literature, were absent from health state descriptions. Most notably, fatigue was cited in over half of the qualitative studies and the following are two examples of text coded under this category:

“Such a small and simple thing that you used to do in no time at all without even thinking, you, well, you now have to put all your energy into it... and also when you have to carry something in, you sort of feel how useless it is (I, male 59 years, married)” [53]

“This feeling of fatigue, it comes as quick as a bolt of lightning. I don’t feel any signals, and all of a sudden I’m totally exhausted. I should have a timer that tickled me every hour, so I know that I should stop and take a rest” [48]

Within the mood and cognition domain, emotional difficulties, dissociation of self and body, and perplexity were absent from all health state descriptions, despite their prominence in the patient perspective. In particular, emotion was the most cited category from qualitative studies, and one of the most cited categories overall in the qualitative literature, whereas expressive problems were most often cited in the health state descriptions. Examples of text coded under `emotion’:

“I thought that it could not be true! I felt desperate because of what had happened. I thought that it could not be true, so I tried to walk, but I couldn’t. . . . I became very depressed and cried a lot.” [42]

Overall, an alphabet of feelings was mentioned: angry, ashamed, bewildered, burdensome, depressed, frustrated, helpless, inadequate, imperfect, shocked, suicidal, surprised, tearful, tetchy, traumatized, vulnerable, worried:

“ this is why I’m so frustrated - everything I do, I’m so slow to what I used to be.” [60]

Within the prognosis domain, concerns about recovery and further risk were prominent in the patient perspective but missing from the health state descriptions. In addition, a wider range of categories relevant to patient experience in the social domain were absent from health state descriptions than any other domain. These included unpredictability and unreliability, concern for next of kin, perception by others, support and networks, and changes in self-identity and social role; the last two were cited most often in qualitative studies, whereas dependence was dominant in the health state descriptions.

Discussion

To our knowledge, this is the first study to explore the extent to which stroke health state descriptions used in preference elicitation studies reflect patients’ experience, by reviewing and comparing published health state descriptions used in elicitation studies with a qualitative synthesis of stroke patients’ perspectives on what is important to them. Variation exists in the content of health state descriptions for all levels of stroke severity. Of greater concern is the major disjuncture between how stroke is experienced by patients and the representation of stroke in the health state descriptions, which appears to reflect an absence of engagement of stroke patient perspectives in their development. This raises significant concerns about the validity of the descriptions and hence the values derived from them, and about the methods used to develop health state descriptions. This has potentially significant consequences for the use of the values elicited using such descriptions in health policy and clinical/patient decisions.

There are some study limitations. Despite covering a range of different strokes and stroke severities [24],[48],[55],[60],[61],[63], the available accounts tend to exclude the perspectives of those with more severe strokes, particularly involving speech impairments [48]. Nonetheless, our review included a wide range of studies and patient groups. In order to indicate the spread of categories across the data sources, we counted the number of data sources (published qualitative studies and health state descriptions) in which these categories appear. This is an indirect measure of importance to patients, but nonetheless captures the presence of key categories across a range of studies exploring patient experience. An inherent limitation of content analysis is that counts of content cannot in itself produce a deep understanding of the data, but to ameliorate this we also conducted and report a thematic synthesis.

Previous research regarding the impact of the measurement process on utility values exists. Specifically for stroke, variations in estimates resulting from direct preference elicitation have previously been explained by a number of factors. First, the choice of study population; for example, healthy participants assign lower utilities than patients who have experienced stroke [11],[17]. Second, the method of elicitation, with standard gamble derived scores being generally higher than those derived from time trade off studies, which in turn are higher than those derived from visual analogue scales [16],[18]. Third, the bounds of the scale, typically 0-1, may be defined differently: the upper bound defined as perfect health yields lower utility scores than if defined as the absence of the condition of interest, which is no guarantee that health is perfect [19].

Variations in health state description content may also explain differences. The level of detail in health state descriptions can affect utility scores; longer, more comprehensive, descriptions (although appearing to have more face validity), may overload respondents’ cognitive capacity, so that they latch onto a few key phrases and ignore the rest [20]. Naming or labelling a condition may have an impact [10],[12],[21]-[23] with a recent study recommending avoiding condition labels in health state descriptions to ensure that values are not affected by prior knowledge or preconceptions of the condition that may distort the health state being valued [10]. Furthermore, the wording may also cause variation in values if not presented in a balanced manner (framing bias), with both positive and negative effects described; explicit inclusion of negative aspects of stroke elicit lower values [24],[25].

A key element in developing valid health state descriptions is whether the description accurately reflects patient experience. However, there is little empirical work on the content validity of health state descriptions, nor on the methods of their development, despite long standing arguments for this [22],[66]; this may explain significant variations or biases in utility scores [67]. This failure to take account of patient experience is the most probable explanation for the observed variation in stroke health state descriptions, and most importantly for the disjuncture between them and what is important to patients. At a population level, variations or biases in utility scores may have serious implications for resource allocation decisions within health care systems. For example, NICE recommends the use of Quality Adjusted Life Years (QALYs) as a measure of health benefit for their `reference case’, to enable a standardized approach for comparing economic evaluations across different healthcare areas [1]. Indirect preference elicitation using the EQ-5D is the method and measure of HRQOL in adults that is preferred by NICE decisions taken at a national level. Despite this, a review of the selection and use of health-related utility values for economic models included in NICE Technology Appraisals [68] found that only 56% of submissions to NICE and assessment reports included utility values that met the relevant reference case. This highlights variation in the methods used to select and incorporate utility values in economic models for NICE Technology Appraisals. Furthermore, methods for guideline production in other countries are in general less prescriptive regarding methods of preference elicitation [69], thus being more likely to incorporate direct methods of preference elicitation. The use of direct preference elicitation methods where health state descriptions may lack content validity could have an impact on the estimated cost effectiveness of health interventions and associated resource allocation decisions.

Health state descriptions were commonly derived from the Modified Rankin Scale (mRS), a clinician-derived measure of global disability [70], but the content of these descriptions differed across studies. Descriptions in other studies were either derived directly from clinicians or their derivation was not described. Most critically, none of the preference elicitation studies utilised stroke patients’ perspectives to inform the content of the descriptions, in stark contrast to the standard methods of developing HRQOL measures [71], where the extent to which patient experience has generated the content and domains is a critical indicator of validity [18],[19].

These findings are disturbing - nearly two decades ago it was recommended that health state description development should draw upon a range of perspectives, by collating information on health states by using the evidence base, and/or interviewing medical professionals and patients, in order to seek a consensus on the most important aspects of quality of life and their relative importance [72]. Similarly, the importance of incorporating patients’ perspectives to establish domain and content validity of the impact of stroke on QOL using qualitative research, either by asking patients directly or by using the evidence base has also been recognised [9]. Despite these recommendations, none of the preference elicitation studies for stroke used these methods.

Given this absence of the patient perspective, the disjuncture between how stroke is experienced and the representation of stroke in the health state descriptions is not surprising. As detailed in the results above, for example, the qualitative studies demonstrate that unresponsive or unpredictably responsive limbs, fatigue, cognitive difficulties and emotional ability led those recovering from stroke to curtail their activities, resulting for many in lives that little resembled their pre-stroke existence. This contrasts with the lack of emphasis on these features in health state descriptions. Furthermore, the effects of stroke on everyday life had significant implications for social role, identity and relationships. No longer able or confident in their ability to engage in everyday activities, the relationships associated with these activities shifted. Stroke survivors often disengaged from, or restricted, their social networks, leading to social isolation. Social relationships were disrupted, via dependence on others, resulting from the physical and emotional effects of stroke. These social characteristics were noticeably absent from the health state descriptions.

Whilst direct methods of preference elicitation have an important role to play, the content validity of health state descriptions for stroke, as with HRQOL instruments, can only be established if patients’ perspectives on the impact of the health state are incorporated into their development. The fact that patients’ experiences of stroke incorporate a much wider set of categories than those incorporated in the health state descriptions, emphasises the importance of this.

Conclusions

Key features of the methodological process for directly eliciting utility values for stroke can explain variations in estimates. One such feature is the design and content of health state descriptions. Our findings not only show that there is no systematic method for constructing the content/scope of health state descriptions for stroke, but also critically that the perspective of patients is not incorporated. We have demonstrated that this produces descriptions with major deficits in reflecting the lived experience of stroke, and raises serious questions about the values derived from such descriptions, which might lead to erroneous conclusions in decisions made based on these values.

We recommend that health state descriptions used for direct preference elicitation, as with HRQOL instruments, should be developed with reference to patient perspectives derived from published qualitative research and/or directly from patients themselves. Further research into the differing impact of descriptions that do or do not incorporate what is important to patients would help to characterise the impact of these deficits in terms of utility scores and associated Quality Adjusted Life Years (QALYs) and resource allocation decisions.

Ethics statement

Ethical approval was not needed for this study as it is based upon two literature reviews and involves no human contact.

Authors’ contributions

JG - Study concept and design, acquisition of data, analysis and interpretation, critical revision of the manuscript for important intellectual content, study supervision. Dr. L - acquisition of data, analysis and interpretation, critical revision of the manuscript for important intellectual content,Prof M - Study concept and design, analysis and interpretation, critical revision of the manuscript for important intellectual content, study supervision. Prof F - critical revision of the manuscript for important intellectual content, study supervision. Dr. M - analysis and interpretation, critical revision of the manuscript for important intellectual content. Prof T - Study concept and design, analysis and interpretation, critical revision of the manuscript for important intellectual content, study supervision. All authors read and approved the final manuscript.