Measuring social difficulties in routine patient-centred assessment: a Rasch analysis of the social difficulties inventory
- First Online:
- Cite this article as:
- Smith, A.B., Wright, P., Selby, P. et al. Qual Life Res (2007) 16: 823. doi:10.1007/s11136-007-9181-9
- 123 Views
Background Social difficulties may add to the psychological burden experienced by cancer patients. Therefore identifying social difficulties in routine oncology practice may help prevent or alleviate distress. The Social Difficulties Inventory (SDI) is a short questionnaire developed for assessing social difficulties in cancer patients. Although well-validated, not enough is known about the clinical meaning and utility of the instrument or whether the items can be meaningfully summed to form a summary index of “Social Distress”. Purpose To determine, using Rasch analysis, whether the SDI could be used as a summary index of social distress specifically examining three fundamental criteria: item fit, unidimensionality and item invariance. Methods The Partial Credit Model was applied to a heterogeneous sample of cancer patients (n = 609) who had completed the SDI. Results Five items were identified as misfitting (infit mean square ≥ 1.3 and standardised t-statistic ≥ 2) and excluded from the subsequent analysis. The remaining items formed a unidimensional interval scale with no additional factors identified in a principal components analysis of the residuals. No differential item functioning was observed for age, gender, diagnosis, extent of disease or social deprivation. The 16-item SDI can be summed to produce an overall index of social distress, facilitating routine identification of social difficulties. Subsequent work is needed to evaluate whether the instrument is able to identify patients with high levels of social distress requiring intervention.
KeywordsSocial distressSocial difficulties inventoryCancerRasch
The social difficulties of cancer patients are broad ranging and are experienced by people of all ages, both sexes, at different stages of disease [1, 2] and from all socio-economic backgrounds . Social difficulties related to cancer and cancer treatments may add to the physical burden, psychological distress and uncertainty already experienced by patients and their families. The biopsychosocial model of health  stresses the importance of seeing the health care issues of patients in their social context. This model has been adopted by the World Health Organization International Classification of Functioning, Disability and Health (ICF) in order to provide a coherent view of different perspectives of health . To meet the needs of patients for supportive and palliative care, including this wider perspective, the National Institute for Clinical Excellence (NICE) recommended assessment and discussion of patients’ needs for physical, psychological, social, spiritual and financial support and specifically, recommends the social care needs of cancer patients are identified and addressed on an ongoing basis . Once these issues are identified the health care team may discuss them with the patients and, if appropriate, suggest interventions with the aim of alleviating the difficulty. This may include, for instance, the provision of practical support, financial and employment advice, and support for carers . However, there are barriers to routine assessment of social care needs in oncology practice which include not only logistical problems of how to assess so many people at different time points, but also of finding an assessment tool that is brief, covers the range of social difficulties of importance to all cancer patients and that is easy to use and score. Advances in electronic data capture methods have provided a way of overcoming the logistical problems of routine patient assessment [7–9].
A 21 item Social Difficulties Inventory (SDI), developed specifically for use in routine oncology practice, fulfils criteria of brevity, simplicity, relevance and practical significance. The inventory covers a broad range of social difficulties, including caring for dependents, problems with financial services and with work, as well as body image issues, sexual matters and personal care. Psychometric evaluation of the 21 individual items demonstrated moderate to very good criterion validity and test-retest reliability. Common factor analysis revealed a simple four-factor structure, three factors being clearly defined and demonstrating good internal reliability “Physical Ability” (5 items), “Contact with Others” (3 items) and “Providing for the family” (5 items). The model explained 46% of the variance only. Summated scales derived from the three factors were used to test and supported construct validity. As the primary purpose of the SDI is to identify people experiencing a range of social difficulties a decision was made to preserve the 21 items to maintain the scope and content validity of the instrument despite the low amount of variance explained by the model . Further research focussed on determining the clinical meaning of the SDI and how it can be used in routine clinical practice.
Using standard psychometric analysis to derive three sub-scale scores may not be the best approach in clinical practice where a quick and simple way of highlighting patients most at need is necessary. Therefore, we decided to explore an alternative strategy based on highlighting individual items with high scores in addition to screening for patients with a high overall score of social distress, but possibly not scoring at the highest level on any individual items. In order to employ such a strategy it is necessary to provide a meaningful summary index of SDI, reflecting a hypothetical single underlying latent trait of social distress. Innovative analytical strategies to health measurement (Rasch models) can provide the mathematical means of testing the assumption of a single underlying latent trait and can be used to improve the measurement ability of the instrument . Rasch models were originally developed for use in education, but are now increasingly used in health and related disciplines, and in particular in cancer [12–14].
The aim of this study was to determine, using Rasch analysis, whether the SDI could be used as a summary index of social distress and specifically verify if the instrument respects three fundamental criteria for measurement: unidimensionality, item fit, and item invariance. Additionally, an item map, as well as the person separation index were used to evaluate the properties of the instrument.
Participants and data collection
This analysis is based on patient data collected from three separate studies: one study evaluating psychometric properties (n = 270) and two studies evaluating its clinical meaning and utility (cross-sectional study n = 189, longitudinal n = 150). Adult cancer patients were recruited from clinics and wards of haematology, oncology, chest medicine and surgical specialities across the Leeds Cancer Centre and asked to complete a number of questionnaires, including the SDI. Clinical and socio-demographic data were collected from medical notes. To be eligible patients had to be able to read English and be physically and mentally able to complete questionnaires using a computer touchscreen. Patients provided written informed consent. The studies had approval from local research ethics committees. Details of the psychometric study [9, 10] and the other studies are published elsewhere .
The social difficulties inventory (SDI)
The SDI is a 21-item questionnaire designed to assess social difficulties experienced by cancer patients over the preceding month. Items are rated on a 4-point scale (0 = no difficulty, 1 = a little difficulty, 2 = quite a bit of difficulty and 3 = very much difficulty). The social difficulties include the following wide ranging items: independence, domestic chores, personal care, care of dependents, support for those close to you, welfare benefits, finances, financial services, work, planning the future, communication with those close to you, communication with others, plans to have a family, sexual matters, body image, isolation, getting around, where you live, recreation, holidays and an “any other difficulty” item.
Statistical model & methods
The Chi-square test and t-tests were used to look at differences between participants and non-participants in terms of age, gender, disease stage and deprivation. Two deprivation groups, using the median Carstairs and Morris Index rate as the cut point (derived from census data from the 1991 census ; Census Dissemination Unit ), were created resulting in one more deprived group (−0.77 to 12.44) and the other more affluent group (−4.51 to −0.86).
Rasch models  are latent trait models which provide estimate of item difficulty or location, i.e. how easily an item can be endorsed, as well as an estimate of person ability or measure. Both item location and person measure, which are measured in log-odds or “logits”, are placed along the same latent trait. The strength of Rasch models is that these two estimates are independent of both the sample of individuals and the questions used.
There are three fundamental criteria for Rasch models, namely item fit, unidimensionality, and item invariance. These are described in more detail below.
Item fit is concerned with whether the items fit the unidimensional Rasch model and can be assessed through a number of fit statistics. The most commonly used fit statistics are infit and outfit mean square residuals and their normalised or standardised forms, the t-statistics. The mean square fit statistics are based on the chi-square distribution, have an expected value of 1, and can range from 0 to infinity. Items with fit statistics substantially lower or greater than one may be either redundant or adding noise to the latent trait, respectively.
A commonly used metric for identifying possible misfitting items is a range of 0.7–1.3 for the infit mean square . However, despite its widespread use in research, there is some evidence from simulated data sets that the Type I error rate (α = 0.05) for the mean square fit statistics may not hold as sample size increases, and some authors recommend that the range is reduced in proportion to sample size . An alternative form of fit statistic, the standardised t-statistic, is the cube root (Wilson–Hilferty) transformation of the chi-square mean squares to an approximate standardised normal distribution . There is some evidence from simulated data sets that the standardised t-statistics may be more effective at highlighting misfitting items. Standardised t-statistics are usually evaluated against ±2 .
The advantage of Rasch models is that when a unidimensional scale has been established an interval scale is produced, where differences between adjacent scores on a scale are equally spaced. This has important implications for measurement, since this allows meaningful comparisons to be made of changes in raw scores of equal dimensions along the latent trait .
Unidimensionality can be used to assess whether the single latent trait explains all the variance in the data. In addition to item fit, unidimensionality can be evaluated through a principal components analysis (PCA) of the residuals when the initial latent trait (i.e. the “Rasch” factor) has been extracted . Additional factors with eigenvalues <1.7 are normally not considered to constitute a threat to unidimensionality . However, there has been some concern expressed in the literature that fit statistics and principal components analyses may not be sufficient to identify multidimensionality [22, 23]. Smith  recommends calibrating item difficulties separately for the entire instrument and then for those items misfitting. An average difference or “shift constant” is calculated, which is used to weight the person measures estimated from the misfitting items alone. Independent paired t-tests can then be performed between the two forms of person measures, and the percentage falling outside the 95% confidence interval can be calculated.
A further feature of Rasch, which is increasingly being exploited in health research, particularly in comparisons of multilingual versions of questionnaires, is the facility to identify item bias or differential item functioning .
Rasch models require the item estimation to be independent of the subgroups of individuals completing the questionnaires. In other words, item parameters should be invariant across populations . Items not demonstrating invariance are commonly referred as exhibiting differential item functioning (DIF) or item bias. Identification of differential item functioning (DIF) allows comparisons and evaluations to be made of whether items are functioning equivalently across important categories, such as diagnosis, extent of disease, and in the context of social difficulties, in social deprivation. Item invariance  can be assessed by producing independent estimates of item location using subgroups of individuals (e.g. groups defined by gender, age group, diagnosis etc.). Differences between item locations are then evaluated using t-tests.
Rasch analysis of SDI data
The data were analysed using Winsteps software . The Partial Credit Model  was applied to the data. This model is appropriate for analysing polytomous items  and treats each item within a questionnaire as an individual scale. This allows the thresholds between response categories to be evaluated for each item, as well as testing the model assumption of frequency of endorsement of categories. Disordering of thresholds and categories may arise where categories are not as well endorsed as assumed. Disordered thresholds and categories may affect fit and consequently categories that are disordered should be collapsed and the Rasch model re-applied to the data.
Ordering of the thresholds and categories
Therefore, the first step in the Rasch analysis was to determine whether disordering of thresholds or categories had occurred for each of the 21 items. Categories were collapsed for any items identified as having disordered thresholds.
Following this the data were re-analysed and item misfit was identified. According to the algorithm advocated by Smith and colleagues (p. 78, Smith et al. ) the infit mean square value needed to identify misfitting items should be 1.08 to account for a sample size of approximately 600 patients. However, this restrictive requirement may be counter-productive leading to a large number of items mistakenly classified as misfitting. Therefore, alongside the commonly employed value for infit mean squares of >1.3 [13, 17], we also adopted the standardised t-statistics of ±2 . Items demonstrating infit mean square statistics of ≥1.3 and infit standardised t-statistics of ≥2 were labelled as misfitting. Misfitting items were removed from the analysis and the Rasch model was re-applied, and fit re-evaluated. This iterative process continued until no further misfit was identified in the fit statistics.
Following on from the analysis of misfitting items unidimensionality was further assessed through a principal components analysis (PCA) of residuals using a criterion of eigenvalues >3.0 accounting for more than 5% of the unexplained variance for additional factors . Additionally, the presence of multidimensionality was assessed using the processed described by Smith .
Item invariance was then evaluated for gender, age groups (<41; 41–60; and >60 years of age), extent of disease (disease free diagnosed less than two years ago, disease free diagnosed more than two years ago “survivors”, primary local, local recurrent, metastatic, and other), diagnosis (11 categories), and the Carstairs–Morris index (2 categories, “affluent” and “deprived”). A Bonferroni adjustment was applied to the critical value to account for multiple testing. DIF was therefore evaluated at a critical value of .001 and a difference in item locations of 0.5 logits .
Item map and person separation
An item-map detailing the locations of the items and distribution of patients’ scores was plotted for the final group of items, as well as the person separation measure (the Rasch measure of internal reliability). These were used to evaluate the final properties of the instrument.
Clinical details of the consenting group
Head and neck
Stage of disease
Disease free diagnosed more than 2 years ago (survivors)
Disease free diagnosed less than 2 years ago
Primary local disease
Local recurrent disease
Rasch analysis of the SDI data
Ordering of thresholds and categories
Disordered categories were observed for items 3 (“Personal Care”), 4 (“Caring for dependents”), 8 (“Financial services”), 9 (“Work”) 14 (“Plans to have a family”) and 18 (“Where you live”). Items 3, 4, 8, 9 and 18 were recoded to a 3 category response scale by collapsing the third and fourth categories (“quite a bit of difficulty” and “very much difficulty”), whereas item 14 was rescored as a dichotomous item (by collapsing the second, third and fourth categories).
Item fit, Item locations and loadings for the 16-item SDI
Caring for dependents
Support for dependents
Planning the future
Communicating with those close
Communicating with others
The variance attributable to the Rasch factor amounted to 72.2%. The final column in Table 2 shows the loadings onto the first factor of the Principal Components Analysis. The largest additional factor found in the residuals had an eigenvalue of 2.6 and accounted for 4.5% of the unexplained variance. Furthermore, only seven pairs of person measures or 1.15% fell outside the 95% confidence interval. It can therefore be concluded that no further factors were present in the data.
The results from the item invariance analysis revealed no differential item functioning (DIF) for any of the categories (data not shown), i.e. none of the items exceeded the criteria of t ≥ 1.96 and a logit difference of 0.5 logits.
Item map and person separation
The person separation measure was 3.01 (equivalent to Cronbach’s alpha of 0.87) demonstrating very high levels of internal reliability. This is demonstrated by the overlap at lower thresholds (e.g. threshold between “No” and “A little”, as well as “A little” and “Quite a bit”) and the person measures (Fig. 1).
Score to logit conversion
The SDI was developed for use in routine clinical practice to identify patients experiencing social difficulties so that issues of concern may be discussed and appropriate support planned. This use relies on staff focussing on items scored at higher levels, “flagged items”, and may result in overlooking a general background of social distress generated from a larger number of lower scoring items. A Rasch analysis was used to explore the measurement properties of the SDI in order to determine whether the SDI could be used as a summary index . The analysis revealed that five items did not fit the model. These items were removed from the analysis, and the remaining 16-item summary index was shown to be a unidimensional structure with no additional factors present. Furthermore, no differential item functioning was observed for age group, gender, diagnostic group, extent of disease or social deprivation suggesting that the instrument functions equivalently for patients when grouped in this way. Therefore the Rasch analysis of the SDI provided the mathematical rationale for generating a single summary index of SDI, using the 16 items that fit a unidimensional latent trait of social distress.
This result has important implications for the use of the SDI in routine clinical practice. The 16-items from the SDI can be summed to form a summary index of “Social Distress”, which will greatly facilitate screening and identification of social difficulties experienced by cancer patients. Furthermore, since this instrument forms an interval scale over almost two thirds of the raw scores it becomes not only possible for changes in social difficulties to be recorded, but it also potentially allows the interpretation of meaningful change over time. Within routine practice the 16-item summary index will provide an overall picture of patients’ social distress, to which high scoring individual items will be added as indicators of specific areas of concern. The 5 items not fitting the model (“Sexual matters”, “Plans to have a family”, “Where you live”, “Holidays” and “Other”), endorsed strongly in earlier stages of the instruments development by staff and patients and that met preset criteria of reliability and validity, will be included and used as single items to provide a complete picture of everyday difficulties experienced.
The Rasch analysis demonstrated that five of the items retained needed to be recoded from a 4-point scale to a 3-point scale. Similarly, the response categories for one further item (“Plans to have a family”) had to be dichotomised. Rather than recommending changes to the instrument, we would suggest that response categories are rescored after respondents have completed the questionnaire in order to calculate the summary index and individual item scores. For instance, based on these results it is recommended that the 16 items are summed to form a single index of Social Distress with scores ranging from 0 to 44 (a scoring template is provided in Appendix 1). The five misfitting items should be scored and interpreted separately as individual items.
From Fig. 1 the distribution of the items and the people along the item map demonstrates that most people report little or no social distress. Such a distribution is appropriate for a screening instrument designed to detect higher levels of distress requiring intervention. For those who indicate social distress this instrument may now be used with confidence across the disease trajectory for any patients with different stages of disease, age and gender in line with the NICE  guidance for social care needs of cancer patients to be identified and addressed on an ongoing basis.
The study sample comprised a wide range of cancer patients with different diagnoses and stages of disease and treatment. However, some limitations should be reported. In all three studies the participation rate for more affluent people was higher than for the less affluent. Although this may lead to an under-representation of deprived patients this may not be of great significance. Analysis has shown that the social impact of cancer tends to overwhelm the impact of deprivation for patients while they are at “active” stages of disease. It is only when people become survivors, a time when social difficulties tend to diminish, that deprivation begins to play a significant role . Furthermore, the lack of any differential item bias by social deprivation demonstrates that the instrument would be sufficiently robust to identify any real differences in the effects of deprivation where they arise.
This is part of an ongoing body of work exploring the clinical meaning and utility of the SDI. Although, a factor analysis of a previous version of the questionnaire has  demonstrated a three-factor structure for the instrument, the Rasch analysis suggests that there is an underlying common, single construct of Social Distress. This important finding provides the mathematical basis for developing a strategy for interpretation and use of the SDI in clinical practice. A single summary index can be used to screen patients with high overall levels of social distress. This will be reported in addition to high scoring individual items to indicate specific areas where help may be needed. On this basis guidelines for staff can be developed on how to interpret the scores and which patients may benefit from discussion and support concerning social issues.
Clearly further clinical validation of this strategy is required. This will be done, in part, by comparing the Rasch person scores to levels of social distress categorised from interviews with a social worker to see if “case level” social distress can be identified. During the validation, the contribution of the earlier identified three sub-scales will be explored as well. Future work will include assessment of the meaning of changes in individual SDI items and general social distress over time.
The authors would like to express their thanks to the patients who completed the questionnaires and interviews, and participated in the focus groups, as well as the team of research assistants who collected the data.