Quality of Life Research

, Volume 16, Issue 5, pp 823–831

Measuring social difficulties in routine patient-centred assessment: a Rasch analysis of the social difficulties inventory

Authors

    • Psychosocial Oncology and Clinical Practice Research Group, Cancer Research UK Clinical Centre in LeedsSt James’s University Hospital
  • Penny Wright
    • Psychosocial Oncology and Clinical Practice Research Group, Cancer Research UK Clinical Centre in LeedsSt James’s University Hospital
  • Peter Selby
    • Psychosocial Oncology and Clinical Practice Research Group, Cancer Research UK Clinical Centre in LeedsSt James’s University Hospital
  • Galina Velikova
    • Psychosocial Oncology and Clinical Practice Research Group, Cancer Research UK Clinical Centre in LeedsSt James’s University Hospital
Original Paper

DOI: 10.1007/s11136-007-9181-9

Cite this article as:
Smith, A.B., Wright, P., Selby, P. et al. Qual Life Res (2007) 16: 823. doi:10.1007/s11136-007-9181-9
  • 123 Views

Abstract

Background Social difficulties may add to the psychological burden experienced by cancer patients. Therefore identifying social difficulties in routine oncology practice may help prevent or alleviate distress. The Social Difficulties Inventory (SDI) is a short questionnaire developed for assessing social difficulties in cancer patients. Although well-validated, not enough is known about the clinical meaning and utility of the instrument or whether the items can be meaningfully summed to form a summary index of “Social Distress”. Purpose To determine, using Rasch analysis, whether the SDI could be used as a summary index of social distress specifically examining three fundamental criteria: item fit, unidimensionality and item invariance. Methods The Partial Credit Model was applied to a heterogeneous sample of cancer patients (n = 609) who had completed the SDI. Results Five items were identified as misfitting (infit mean square ≥ 1.3 and standardised t-statistic ≥ 2) and excluded from the subsequent analysis. The remaining items formed a unidimensional interval scale with no additional factors identified in a principal components analysis of the residuals. No differential item functioning was observed for age, gender, diagnosis, extent of disease or social deprivation. The 16-item SDI can be summed to produce an overall index of social distress, facilitating routine identification of social difficulties. Subsequent work is needed to evaluate whether the instrument is able to identify patients with high levels of social distress requiring intervention.

Keywords

Social distressSocial difficulties inventoryCancerRasch

Introduction

The social difficulties of cancer patients are broad ranging and are experienced by people of all ages, both sexes, at different stages of disease [1, 2] and from all socio-economic backgrounds [3]. Social difficulties related to cancer and cancer treatments may add to the physical burden, psychological distress and uncertainty already experienced by patients and their families. The biopsychosocial model of health [4] stresses the importance of seeing the health care issues of patients in their social context. This model has been adopted by the World Health Organization International Classification of Functioning, Disability and Health (ICF) in order to provide a coherent view of different perspectives of health [5]. To meet the needs of patients for supportive and palliative care, including this wider perspective, the National Institute for Clinical Excellence (NICE) recommended assessment and discussion of patients’ needs for physical, psychological, social, spiritual and financial support and specifically, recommends the social care needs of cancer patients are identified and addressed on an ongoing basis [6]. Once these issues are identified the health care team may discuss them with the patients and, if appropriate, suggest interventions with the aim of alleviating the difficulty. This may include, for instance, the provision of practical support, financial and employment advice, and support for carers [6]. However, there are barriers to routine assessment of social care needs in oncology practice which include not only logistical problems of how to assess so many people at different time points, but also of finding an assessment tool that is brief, covers the range of social difficulties of importance to all cancer patients and that is easy to use and score. Advances in electronic data capture methods have provided a way of overcoming the logistical problems of routine patient assessment [79].

A 21 item Social Difficulties Inventory (SDI), developed specifically for use in routine oncology practice, fulfils criteria of brevity, simplicity, relevance and practical significance. The inventory covers a broad range of social difficulties, including caring for dependents, problems with financial services and with work, as well as body image issues, sexual matters and personal care. Psychometric evaluation of the 21 individual items demonstrated moderate to very good criterion validity and test-retest reliability. Common factor analysis revealed a simple four-factor structure, three factors being clearly defined and demonstrating good internal reliability “Physical Ability” (5 items), “Contact with Others” (3 items) and “Providing for the family” (5 items). The model explained 46% of the variance only. Summated scales derived from the three factors were used to test and supported construct validity. As the primary purpose of the SDI is to identify people experiencing a range of social difficulties a decision was made to preserve the 21 items to maintain the scope and content validity of the instrument despite the low amount of variance explained by the model [10]. Further research focussed on determining the clinical meaning of the SDI and how it can be used in routine clinical practice.

Using standard psychometric analysis to derive three sub-scale scores may not be the best approach in clinical practice where a quick and simple way of highlighting patients most at need is necessary. Therefore, we decided to explore an alternative strategy based on highlighting individual items with high scores in addition to screening for patients with a high overall score of social distress, but possibly not scoring at the highest level on any individual items. In order to employ such a strategy it is necessary to provide a meaningful summary index of SDI, reflecting a hypothetical single underlying latent trait of social distress. Innovative analytical strategies to health measurement (Rasch models) can provide the mathematical means of testing the assumption of a single underlying latent trait and can be used to improve the measurement ability of the instrument [11]. Rasch models were originally developed for use in education, but are now increasingly used in health and related disciplines, and in particular in cancer [1214].

Aim

The aim of this study was to determine, using Rasch analysis, whether the SDI could be used as a summary index of social distress and specifically verify if the instrument respects three fundamental criteria for measurement: unidimensionality, item fit, and item invariance. Additionally, an item map, as well as the person separation index were used to evaluate the properties of the instrument.

Method

Participants and data collection

This analysis is based on patient data collected from three separate studies: one study evaluating psychometric properties (n = 270) and two studies evaluating its clinical meaning and utility (cross-sectional study n = 189, longitudinal n = 150). Adult cancer patients were recruited from clinics and wards of haematology, oncology, chest medicine and surgical specialities across the Leeds Cancer Centre and asked to complete a number of questionnaires, including the SDI. Clinical and socio-demographic data were collected from medical notes. To be eligible patients had to be able to read English and be physically and mentally able to complete questionnaires using a computer touchscreen. Patients provided written informed consent. The studies had approval from local research ethics committees. Details of the psychometric study [9, 10] and the other studies are published elsewhere [3].

The social difficulties inventory (SDI)

The SDI is a 21-item questionnaire designed to assess social difficulties experienced by cancer patients over the preceding month. Items are rated on a 4-point scale (0 = no difficulty, 1 = a little difficulty, 2 = quite a bit of difficulty and 3 = very much difficulty). The social difficulties include the following wide ranging items: independence, domestic chores, personal care, care of dependents, support for those close to you, welfare benefits, finances, financial services, work, planning the future, communication with those close to you, communication with others, plans to have a family, sexual matters, body image, isolation, getting around, where you live, recreation, holidays and an “any other difficulty” item.

Statistical model & methods

Sample characteristics

The Chi-square test and t-tests were used to look at differences between participants and non-participants in terms of age, gender, disease stage and deprivation. Two deprivation groups, using the median Carstairs and Morris Index rate as the cut point (derived from census data from the 1991 census [15]; Census Dissemination Unit [16]), were created resulting in one more deprived group (−0.77 to 12.44) and the other more affluent group (−4.51 to −0.86).

Rasch analysis

Rasch model

Rasch models [11] are latent trait models which provide estimate of item difficulty or location, i.e. how easily an item can be endorsed, as well as an estimate of person ability or measure. Both item location and person measure, which are measured in log-odds or “logits”, are placed along the same latent trait. The strength of Rasch models is that these two estimates are independent of both the sample of individuals and the questions used.

There are three fundamental criteria for Rasch models, namely item fit, unidimensionality, and item invariance. These are described in more detail below.

Item fit

Item fit is concerned with whether the items fit the unidimensional Rasch model and can be assessed through a number of fit statistics. The most commonly used fit statistics are infit and outfit mean square residuals and their normalised or standardised forms, the t-statistics. The mean square fit statistics are based on the chi-square distribution, have an expected value of 1, and can range from 0 to infinity. Items with fit statistics substantially lower or greater than one may be either redundant or adding noise to the latent trait, respectively.

A commonly used metric for identifying possible misfitting items is a range of 0.7–1.3 for the infit mean square [17]. However, despite its widespread use in research, there is some evidence from simulated data sets that the Type I error rate (α = 0.05) for the mean square fit statistics may not hold as sample size increases, and some authors recommend that the range is reduced in proportion to sample size [18]. An alternative form of fit statistic, the standardised t-statistic, is the cube root (Wilson–Hilferty) transformation of the chi-square mean squares to an approximate standardised normal distribution [18]. There is some evidence from simulated data sets that the standardised t-statistics may be more effective at highlighting misfitting items. Standardised t-statistics are usually evaluated against ±2 [19].

The advantage of Rasch models is that when a unidimensional scale has been established an interval scale is produced, where differences between adjacent scores on a scale are equally spaced. This has important implications for measurement, since this allows meaningful comparisons to be made of changes in raw scores of equal dimensions along the latent trait [20].

Unidimensionality

Unidimensionality can be used to assess whether the single latent trait explains all the variance in the data. In addition to item fit, unidimensionality can be evaluated through a principal components analysis (PCA) of the residuals when the initial latent trait (i.e. the “Rasch” factor) has been extracted [21]. Additional factors with eigenvalues <1.7 are normally not considered to constitute a threat to unidimensionality [21]. However, there has been some concern expressed in the literature that fit statistics and principal components analyses may not be sufficient to identify multidimensionality [22, 23]. Smith [23] recommends calibrating item difficulties separately for the entire instrument and then for those items misfitting. An average difference or “shift constant” is calculated, which is used to weight the person measures estimated from the misfitting items alone. Independent paired t-tests can then be performed between the two forms of person measures, and the percentage falling outside the 95% confidence interval can be calculated.

Item invariance

A further feature of Rasch, which is increasingly being exploited in health research, particularly in comparisons of multilingual versions of questionnaires, is the facility to identify item bias or differential item functioning [24].

Rasch models require the item estimation to be independent of the subgroups of individuals completing the questionnaires. In other words, item parameters should be invariant across populations [19]. Items not demonstrating invariance are commonly referred as exhibiting differential item functioning (DIF) or item bias. Identification of differential item functioning (DIF) allows comparisons and evaluations to be made of whether items are functioning equivalently across important categories, such as diagnosis, extent of disease, and in the context of social difficulties, in social deprivation. Item invariance [21] can be assessed by producing independent estimates of item location using subgroups of individuals (e.g. groups defined by gender, age group, diagnosis etc.). Differences between item locations are then evaluated using t-tests.

Rasch analysis of SDI data

The data were analysed using Winsteps software [21]. The Partial Credit Model [25] was applied to the data. This model is appropriate for analysing polytomous items [19] and treats each item within a questionnaire as an individual scale. This allows the thresholds between response categories to be evaluated for each item, as well as testing the model assumption of frequency of endorsement of categories. Disordering of thresholds and categories may arise where categories are not as well endorsed as assumed. Disordered thresholds and categories may affect fit and consequently categories that are disordered should be collapsed and the Rasch model re-applied to the data.

Ordering of the thresholds and categories

Therefore, the first step in the Rasch analysis was to determine whether disordering of thresholds or categories had occurred for each of the 21 items. Categories were collapsed for any items identified as having disordered thresholds.

Item fit

Following this the data were re-analysed and item misfit was identified. According to the algorithm advocated by Smith and colleagues (p. 78, Smith et al. [18]) the infit mean square value needed to identify misfitting items should be 1.08 to account for a sample size of approximately 600 patients. However, this restrictive requirement may be counter-productive leading to a large number of items mistakenly classified as misfitting. Therefore, alongside the commonly employed value for infit mean squares of >1.3 [13, 17], we also adopted the standardised t-statistics of ±2 [22]. Items demonstrating infit mean square statistics of ≥1.3 and infit standardised t-statistics of ≥2 were labelled as misfitting. Misfitting items were removed from the analysis and the Rasch model was re-applied, and fit re-evaluated. This iterative process continued until no further misfit was identified in the fit statistics.

Unidimensionality

Following on from the analysis of misfitting items unidimensionality was further assessed through a principal components analysis (PCA) of residuals using a criterion of eigenvalues >3.0 accounting for more than 5% of the unexplained variance for additional factors [21]. Additionally, the presence of multidimensionality was assessed using the processed described by Smith [23].

Item invariance

Item invariance was then evaluated for gender, age groups (<41; 41–60; and >60 years of age), extent of disease (disease free diagnosed less than two years ago, disease free diagnosed more than two years ago “survivors”, primary local, local recurrent, metastatic, and other), diagnosis (11 categories), and the Carstairs–Morris index (2 categories, “affluent” and “deprived”). A Bonferroni adjustment was applied to the critical value to account for multiple testing. DIF was therefore evaluated at a critical value of .001 and a difference in item locations of 0.5 logits [13].

Item map and person separation

An item-map detailing the locations of the items and distribution of patients’ scores was plotted for the final group of items, as well as the person separation measure (the Rasch measure of internal reliability). These were used to evaluate the final properties of the instrument.

Results

Sample characteristics

Six hundred and nine people consented to take part and 233 refused (participation rate 72.3%). Participants’ characteristics are presented in Table 1. The total number of non-participants was 82 (33 (40%) males and 49 (60%) females). The average age of non-participants was 56.02 years (range 19–85 years) and 61.62 years (range 18–87) for males and females, respectively. People in the consenting group were on average younger (t = −3.723, df = 840, p < .0001), more likely to be men (Chi-square = 5.134, df = 1, p = .023) and more affluent (missing Carstairs and Morris scores = 43) (t  =  −2.548, df = 797, p = .011) than in the refusing group. No differences were found between the groups in terms of disease stage.
Table 1

Clinical details of the consenting group

Age (years)

Mean (SD)

 

56.1 (15.29)

Gender

Number (%)

 Female

305 (50)

 Male

304 (50%)

Diagnostic group

Number (%)

 Breast

104 (17)

 Brain

4 (1)

 Gastro-intestinal

93 (15)

 Genito-urinary

45 (7)

 Germ cell

60 (9)

 Gynaecological

70 (11)

 Haematological

83 (13)

 Head and neck

53 (9)

 Lung

47 (7)

 Melanoma

29 (5)

 Sarcoma

19 (3)

 Unknown primary

2 (<1)

Stage of disease

 Disease free diagnosed more than 2 years ago (survivors)

94 (15)

 Disease free diagnosed less than 2 years ago

176 (29)

 Primary local disease

125 (21)

 Local recurrent disease

22 (4)

 Metastatic disease

143 (23)

 Othera

49 (8)

a Includes people with advance ovarian cancer and haematological malignancies which cannot be classified using other categories

Rasch analysis of the SDI data

Ordering of thresholds and categories

Disordered categories were observed for items 3 (“Personal Care”), 4 (“Caring for dependents”), 8 (“Financial services”), 9 (“Work”) 14 (“Plans to have a family”) and 18 (“Where you live”). Items 3, 4, 8, 9 and 18 were recoded to a 3 category response scale by collapsing the third and fourth categories (“quite a bit of difficulty” and “very much difficulty”), whereas item 14 was rescored as a dichotomous item (by collapsing the second, third and fourth categories).

Item fit

Item misfit (infit mean square ≥1.30 and standardised t-statistic ≥  ±2) was observed for 5 items, namely items 13 (“Sexual matters”), 14 (“Plans to have a family”), 18 (“Where you live”), 20 (“Holidays”) and 21 (“Other”). These items were removed and the data were re-analysed. The subsequent analysis revealed that none of the remaining items demonstrated misfit (Table 2).
Table 2

Item fit, Item locations and loadings for the 16-item SDI

Items

Description

Location

Infit

Factor loading

MNSQ

ZSTD

SDI1

Independence

−0.38

0.82

−1.2

0.48

SDI2

Domestic chores

−1.05

0.82

−1.3

0.60

SDI3

Personal care

−0.44

0.85

−1.1

0.44

SDI4

Caring for dependents

−0.52

0.71

−2.2

0.29

SDI5

Support for dependents

0.31

0.92

−0.4

−0.19

SDI6

Welfare benefits

0.14

1.17

0.8

−0.52

SDI7

Finances

0.36

1.06

0.4

−0.57

SDI8

Financial services

1.21

1.02

0.2

−0.50

SDI9

Work

0.53

1.16

0.8

−0.33

SDI10

Planning the future

0.39

1.1

0.5

−0.48

SDI11

Communicating with those close

0.18

0.99

0

−0.08

SDI12

Communicating with others

0.64

1.12

0.7

−0.24

SDI15

Body image

0.03

1.31

1.6

−0.25

SDI16

Isolation

0.00

1.04

0.3

−0.25

SDI17

Getting around

−0.33

0.95

−0.3

0.47

SDI19

Recreation

−1.07

1.08

0.7

0.29

Unidimensionality

The variance attributable to the Rasch factor amounted to 72.2%. The final column in Table 2 shows the loadings onto the first factor of the Principal Components Analysis. The largest additional factor found in the residuals had an eigenvalue of 2.6 and accounted for 4.5% of the unexplained variance. Furthermore, only seven pairs of person measures or 1.15% fell outside the 95% confidence interval. It can therefore be concluded that no further factors were present in the data.

Item invariance

The results from the item invariance analysis revealed no differential item functioning (DIF) for any of the categories (data not shown), i.e. none of the items exceeded the criteria of t  ≥ 1.96 and a logit difference of 0.5 logits.

Item map and person separation

The item map is shown in Fig. 1. It can be seen from this figure that “Financial services (loans, mortgages, insurance)” (Item 8) was the hardest item to endorse (in other words that higher levels of “social distress” were required before individual patients endorse this question), whereas “Domestic chores” (Item 2) and “Recreation” (Item 19) were the easiest to endorse. The items covered a range from approximately −1 to +1 logits. The mean person measure was low, −2.10 logits, indicating most patients did not indicate significant social distress. The items were targeting higher levels of social difficulties, as appropriate for a screening instrument.
https://static-content.springer.com/image/art%3A10.1007%2Fs11136-007-9181-9/MediaObjects/11136_2007_9181_Fig1_HTML.gif
Fig. 1

Item Map for Items from the SDI

The person separation measure was 3.01 (equivalent to Cronbach’s alpha of 0.87) demonstrating very high levels of internal reliability. This is demonstrated by the overlap at lower thresholds (e.g. threshold between “No” and “A little”, as well as “A little” and “Quite a bit”) and the person measures (Fig. 1).

Figure 2 demonstrates that the revised SDI forms an interval scale over an approximate range of raw scores from 9 to 38. Table 3 provides the raw score to logit conversion for the data.
https://static-content.springer.com/image/art%3A10.1007%2Fs11136-007-9181-9/MediaObjects/11136_2007_9181_Fig2_HTML.gif
Fig. 2

Person scores plotted against person measures (logits)

Table 3

Score to logit conversion

Score

Measure

SE

0

−4.68

1.84

1

−3.45

1.02

2

−2.73

0.72

3

−2.30

0.59

4

−2.00

0.52

5

−1.76

0.46

6

−1.56

0.43

7

−1.39

0.40

8

−1.25

0.37

9

−1.11

0.36

10

−0.99

0.34

11

−0.88

0.33

12

−0.77

0.32

13

−0.67

0.31

14

−0.58

0.30

15

−0.49

0.30

16

−0.40

0.29

17

−0.32

0.29

18

−0.23

0.28

19

−0.16

0.28

20

−0.08

0.28

21

0.00

0.28

22

0.07

0.27

23

0.15

0.27

24

0.22

0.27

25

0.30

0.28

26

0.38

0.28

27

0.45

0.28

28

0.53

0.28

29

0.62

0.29

30

0.70

0.30

31

0.79

0.30

32

0.88

0.31

33

0.99

0.32

34

1.09

0.34

35

1.21

0.35

36

1.34

0.37

37

1.49

0.39

38

1.65

0.43

39

1.86

0.47

40

2.11

0.54

41

2.47

0.66

42

3.09

0.95

43

4.22

1.79

Discussion

The SDI was developed for use in routine clinical practice to identify patients experiencing social difficulties so that issues of concern may be discussed and appropriate support planned. This use relies on staff focussing on items scored at higher levels, “flagged items”, and may result in overlooking a general background of social distress generated from a larger number of lower scoring items. A Rasch analysis was used to explore the measurement properties of the SDI in order to determine whether the SDI could be used as a summary index [9]. The analysis revealed that five items did not fit the model. These items were removed from the analysis, and the remaining 16-item summary index was shown to be a unidimensional structure with no additional factors present. Furthermore, no differential item functioning was observed for age group, gender, diagnostic group, extent of disease or social deprivation suggesting that the instrument functions equivalently for patients when grouped in this way. Therefore the Rasch analysis of the SDI provided the mathematical rationale for generating a single summary index of SDI, using the 16 items that fit a unidimensional latent trait of social distress.

This result has important implications for the use of the SDI in routine clinical practice. The 16-items from the SDI can be summed to form a summary index of “Social Distress”, which will greatly facilitate screening and identification of social difficulties experienced by cancer patients. Furthermore, since this instrument forms an interval scale over almost two thirds of the raw scores it becomes not only possible for changes in social difficulties to be recorded, but it also potentially allows the interpretation of meaningful change over time. Within routine practice the 16-item summary index will provide an overall picture of patients’ social distress, to which high scoring individual items will be added as indicators of specific areas of concern. The 5 items not fitting the model (“Sexual matters”, “Plans to have a family”, “Where you live”, “Holidays” and “Other”), endorsed strongly in earlier stages of the instruments development by staff and patients and that met preset criteria of reliability and validity, will be included and used as single items to provide a complete picture of everyday difficulties experienced.

The Rasch analysis demonstrated that five of the items retained needed to be recoded from a 4-point scale to a 3-point scale. Similarly, the response categories for one further item (“Plans to have a family”) had to be dichotomised. Rather than recommending changes to the instrument, we would suggest that response categories are rescored after respondents have completed the questionnaire in order to calculate the summary index and individual item scores. For instance, based on these results it is recommended that the 16 items are summed to form a single index of Social Distress with scores ranging from 0 to 44 (a scoring template is provided in Appendix 1). The five misfitting items should be scored and interpreted separately as individual items.

From Fig. 1 the distribution of the items and the people along the item map demonstrates that most people report little or no social distress. Such a distribution is appropriate for a screening instrument designed to detect higher levels of distress requiring intervention. For those who indicate social distress this instrument may now be used with confidence across the disease trajectory for any patients with different stages of disease, age and gender in line with the NICE [6] guidance for social care needs of cancer patients to be identified and addressed on an ongoing basis.

The study sample comprised a wide range of cancer patients with different diagnoses and stages of disease and treatment. However, some limitations should be reported. In all three studies the participation rate for more affluent people was higher than for the less affluent. Although this may lead to an under-representation of deprived patients this may not be of great significance. Analysis has shown that the social impact of cancer tends to overwhelm the impact of deprivation for patients while they are at “active” stages of disease. It is only when people become survivors, a time when social difficulties tend to diminish, that deprivation begins to play a significant role [3]. Furthermore, the lack of any differential item bias by social deprivation demonstrates that the instrument would be sufficiently robust to identify any real differences in the effects of deprivation where they arise.

This is part of an ongoing body of work exploring the clinical meaning and utility of the SDI. Although, a factor analysis of a previous version of the questionnaire has [10] demonstrated a three-factor structure for the instrument, the Rasch analysis suggests that there is an underlying common, single construct of Social Distress. This important finding provides the mathematical basis for developing a strategy for interpretation and use of the SDI in clinical practice. A single summary index can be used to screen patients with high overall levels of social distress. This will be reported in addition to high scoring individual items to indicate specific areas where help may be needed. On this basis guidelines for staff can be developed on how to interpret the scores and which patients may benefit from discussion and support concerning social issues.

Clearly further clinical validation of this strategy is required. This will be done, in part, by comparing the Rasch person scores to levels of social distress categorised from interviews with a social worker to see if “case level” social distress can be identified. During the validation, the contribution of the earlier identified three sub-scales will be explored as well. Future work will include assessment of the meaning of changes in individual SDI items and general social distress over time.

Acknowledgements

The authors would like to express their thanks to the patients who completed the questionnaires and interviews, and participated in the focus groups, as well as the team of research assistants who collected the data.

Copyright information

© Springer Science+Business Media B.V. 2007