Skip to main content
SpringerLink
Log in

On the validity of measuring change over time in routine clinical assessment: a close examination of item-level response shifts in psychosomatic inpatients

  • Special Section: Response Shift Effects at Item Level (by invitation only)
  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Objective

Significant life events such as severe health status changes or intensive medical treatment often trigger response shifts in individuals that may hamper the comparison of measurements over time. Drawing from the Oort model, this study aims at detecting response shift at the item level in psychosomatic inpatients and evaluating its impact on the validity of comparing repeated measurements.

Study design and setting

Complete pretest and posttest data were available from 1188 patients who had filled out the ICD-10 Symptom Rating (ISR) scale at admission and discharge, on average 24 days after intake. Reconceptualization, reprioritization, and recalibration response shifts were explored applying tests of measurement invariance. In the item-level approach, all model parameters were constrained to be equal between pretest and posttest. If non-invariance was detected, these were linked to the different types of response shift.

Results

When constraining across-occasion model parameters, model fit worsened as indicated by a significant Satorra–Bentler Chi-square difference test suggesting potential presence of response shifts. A close examination revealed presence of two types of response shift, i.e., (non)uniform recalibration and both higher- and lower-level reconceptualization response shifts leading to four model adjustments.

Conclusions

Our analyses suggest that psychosomatic inpatients experienced some response shifts during their hospital stay. According to the hierarchy of measurement invariance, however, only one of the detected non-invariances is critical for unbiased mean comparisons over time, which did not have a substantial impact on estimating change. Hence, the use of the ISR can be recommended for outcomes assessment in clinical routine, as change score estimates do not seem hampered by response shift effects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Ahmed, S., Berzon, R. A., Revicki, D. A., Lenderking, W. R., Moinpour, C. M., Basch, E., et al. (2012). The use of patient-reported outcomes (PRO) within comparative effectiveness research: Implications for clinical practice and health care policy. Medical Care, 50(12), 1060–1070.

    Article  PubMed  Google Scholar 

  2. Basch, E., Abernethy, A. P., & Reeve, B. B. (2011). Assuring the patient centeredness of patient-reported outcomes: Content validity in medical product development and comparative effectiveness research. Value Health, 14(8), 965–966.

    Article  PubMed  PubMed Central  Google Scholar 

  3. European Medicines Agency. (2005). Reflection paper on the regulatory guidance for the use of health-related quality of life (HRQL) measures in the evaluation of medicinal products. Retrieved February 2, 2015 from http://www.emea.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003637.pdf.

  4. Food and Drug Administration. (2009). Guidance for industry. Patient-reported outcome measures: Use in medical product development to support labeling claims. Retrieved February 2, 2015 from http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf.

  5. Frank, L., Basch, E., & Selby, J. V. (2014). The PCORI perspective on patient-centered outcomes research. JAMA, 312(15), 1513–1514.

    Article  CAS  PubMed  Google Scholar 

  6. Golembiewski, R. T., Billingsley, K., & Yeager, S. (1976). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavioral Science, 12, 133–157.

    Article  Google Scholar 

  7. Howard, G. S., & Dailey, P. R. (1979). Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64(2), 144–150.

    Article  Google Scholar 

  8. Sprangers, M. A. G. (1996). Response-shift bias: A challenge to the assessment of patients’ quality of life in cancer clinical trials. Cancer Treatment Reviews, 22(Suppl 1), 55–62.

    Article  PubMed  Google Scholar 

  9. Sprangers, M. A. G., & Schwartz, C. E. (1999). Integrating response shift into health-related quality of life research: A theoretical model. Social Science and Medicine, 48(11), 1507–1515.

    Article  CAS  PubMed  Google Scholar 

  10. Schmitt, N. (1982). The use of analysis of covariance structure to assess beta and gamma change. Multivariate Behavioral Research, 17(3), 343–358.

    Article  CAS  PubMed  Google Scholar 

  11. Terborg, J., Howard, G., & Maxwell, S. (1980). Evaluating planned organizational change: A method for assessing alpha, beta, and gamma change. Academy Management Review, 5, 109–121.

    Google Scholar 

  12. Howard, G. (1980). Response-shift bias—A problem in evaluating interventions with pre/post self-reports. Evaluation Review, 4, 93–106.

    Article  Google Scholar 

  13. Schwartz, C. E., & Sprangers, M. A. G. (1999). Methodological approaches for assessing response shift in longitudinal health-related quality-of-life research. Social Science and Medicine, 48(11), 1531–1548.

    Article  CAS  PubMed  Google Scholar 

  14. Osborne, R. H., Hawkins, M., & Sprangers, M. A. G. (2006). Change of perspective: A measurable and desired outcome of chronic disease self-management intervention programs that violates the premise of preintervention/postintervention assessment. Arthritis Care & Research, 55(3), 458–465.

    Article  Google Scholar 

  15. Cronbach, L. J., & Furby, L. (1970). How should we measure “change”—Or should we? Psychological Bulletin, 74(1), 68–80.

    Article  Google Scholar 

  16. Schwartz, C. E., Bode, R., Repucci, N., Becker, J., Sprangers, M. A., & Fayers, P. M. (2006). The clinical significance of adaptation to changing health: A meta-analysis of response shift. Quality of Life Research, 15(9), 1533–1550.

    Article  PubMed  Google Scholar 

  17. Schwartz, C., & Sprangers, M. (2010). Guidelines for improving the stringency of response shift research using the thentest. Quality of Life Research, 19, 455–464.

    Article  PubMed  Google Scholar 

  18. Nolte, S., Elsworth, G. R., Sinclair, A. J., & Osborne, R. H. (2009). Tests of measurement invariance failed to support the application of the “then-test”. Journal of Clinical Epidemiology, 62(11), 1173–1180.

    Article  PubMed  Google Scholar 

  19. Schwartz, C. E., & Rapkin, B. D. (2012). Understanding appraisal processes underlying the thentest: A mixed methods investigation. Quality of Life Research, 21, 381–388.

    Article  PubMed  Google Scholar 

  20. Oort, F. J. (2005). Using structural equation modeling to detect response shift and true change. Quality of Life Research, 14(3), 587–598.

    Article  PubMed  Google Scholar 

  21. Oort, F. J., Visser, M. R. M., & Sprangers, M. A. G. (2005). An application of structural equation modeling to detect response shifts and true change in quality of life data from cancer patients undergoing invasive surgery. Quality of Life Research, 14(3), 599–609.

    Article  PubMed  Google Scholar 

  22. Rapkin, B., & Schwartz, C. (2004). Toward a theoretical model of quality-of-life appraisal: Implications of findings from studies of response shift. Health Qual Life Outcomes, 2(1), 14.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Nolte, S., Elsworth, G. R., Newman, S., & Osborne, R. H. (2013). Measurement issues in the evaluation of chronic disease self-management programs. Quality of Life Research, 22(7), 1655–1664.

    Article  PubMed  Google Scholar 

  24. Gibbons, F. X. (1999). Social comparison as a mediator of response shift. Social Science & Medicine, 48(11), 1517–1530.

    Article  CAS  Google Scholar 

  25. Dibb, B., & Yardley, L. (2006). How does social comparison within a self-help group influence adjustment to chronic illness? A longitudinal study. Social Science & Medicine, 63(6), 1602–1613.

    Article  Google Scholar 

  26. Shedler, J. (2010). The efficacy of psychodynamic psychotherapy. American Psychologist, 65(2), 98–109.

    Article  PubMed  Google Scholar 

  27. Johansson, P., Hoglend, P., & Hersoug, A. G. (2011). Therapeutic alliance mediates the effect of patient expectancy in dynamic psychotherapy. British Journal of Clinical Psychology, 50(3), 283–297.

    PubMed  Google Scholar 

  28. Gregorich, S. (2006). Do self-report instruments allow meaningful comparisons across diverse population groups? Testing measurement invariance using the confirmatory factor analysis framework. Medical Care, 44(11), S78–S94.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Tritt, K., von Heymann, F., Zaudig, M., Zacharias, I., Söllner, W., & Loew, T. (2008). Development of the “ICD-10-Symptom-Rating” (ISR) questionnaire. Zeitschrift fur Psychosomatische Medizin und Psychotherapie, 54(4), 409–418.

    Article  PubMed  Google Scholar 

  30. Deutsches Institut für Medizinische Dokumentation und Information. (2012). ICD-10-GM Version 2013 Systematisches Verzeichnis, Internationale statistische Klassifikation der Krankheiten und verwandter Gesundheitsprobleme, 10 Revision, German Modification.

  31. Fischer, H. F., Tritt, K., Klapp, B. F., & Fliege, H. (2010). Factor structure and psychometric properties of the ICD-10-Symptom-Rating (ISR) in samples of psychosomatic patients. Psychotherapie, Psychosomatik, Medizinische Psychologie, 60(8), 307–315.

    Article  PubMed  Google Scholar 

  32. van de Schoot, R., Kluytmans, A., Tummers, L., Lugtig, P., Hox, J., & Muthen, B. (2013). Facing off with Scylla and Charybdis: A comparison of scalar, partial, and the novel possibility of approximate measurement invariance. Frontiers in Psychology, 4, 770.

    PubMed  PubMed Central  Google Scholar 

  33. Meredith, W. (1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58, 525–543.

    Article  Google Scholar 

  34. Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105, 456–466.

    Article  Google Scholar 

  35. Steenkamp, J.-B., & Baumgartner, H. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78–90.

    Article  Google Scholar 

  36. Hair, J. F., Black, W. C., Rabin, B. J., Anderson, R. E., & Tatham, R. L. (2006). Multivariate data analysis (6th ed.). Upper Saddle River, NJ: Pearson Education Inc.

    Google Scholar 

  37. van de Vliert, E., Huismans, S., & Stok, J. (1985). The criterion approach to unraveling beta and alpha change. Academy of Management Review, 10, 269–274.

    Article  Google Scholar 

  38. Satorra, A., & Bentler, P. (2001). A scaled difference Chi square test statistic for moment structure analysis. Psychometrika, 66, 507–514.

    Article  Google Scholar 

  39. Bryant, F., & Satorra, A. (2012). Principles and practice of scaled difference Chi square testing. Structural Equation Modeling: A Multidisciplinary Journal, 19(3), 372–398.

    Article  Google Scholar 

  40. Bryant, F. B., & Satorra, A. (2013). EXCEL macro file for conducting scaled difference Chi-square tests via LISREL 8, LISREL 9, EQS, and Mplus. Available from the authors. http://www.econ.upf.edu/~satorra/dades/BryantSatorraScaledDifferenceTestsForLISREL8LISREL9EQSandMplus.xls.

  41. Jöreskog, K. G., & Sörbom, D. (1996–2001). LISREL 8: User’s reference guide (2nd ed.). Lincolnwood, IL: Scientific Software International.

  42. Jöreskog, K. (1990). New developments in LISREL: Analysis of ordinal variables using polychoric correlations and weighted least squares. Quality & Quantity, 24(4), 387–404.

    Article  Google Scholar 

  43. Jöreskog, K., & Sörbom, D. (1996–2002). PRELIS 2: User’s reference guide. Lincolnwood, IL: Scientific Software International.

  44. Rose, M., Wahl, I., Crusius, J., & Lowe, B. (2011). Psychological comorbidity. A challenge in acute care. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz, 54(1), 83–89.

    Article  CAS  PubMed  Google Scholar 

  45. Oort, F. J. (2005). Towards a formal definition of response shift (in reply to G.W. Donaldson). Quality of Life Research, 14(10), 2353–2355.

    Article  PubMed  Google Scholar 

  46. Donaldson, G. (2005). Structural equation models for quality of life response shifts: Promises and pitfalls. Quality of Life Research, 14, 2345–2351.

    Article  PubMed  Google Scholar 

  47. Muthen, B., & Asparouhov, T. (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17(3), 313–335.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

The authors greatly appreciate the invaluable feedback by Dr Carolyn Schwartz on earlier drafts of the manuscript and the critical review of three anonymous reviewers.

Author information

Authors and Affiliations

  1. Department of Psychosomatic Medicine, Center for Internal Medicine and Dermatology, Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany

    S. Nolte, A. Mierke, H. F. Fischer & M. Rose

  2. Population Health Strategic Research Centre, School of Health and Social Development, Deakin University, Burwood, VIC, 3125, Australia

    S. Nolte

  3. Institute for Social Medicine, Epidemiology and Health Economics, Charité – Universitätsmedizin Berlin, Berlin, Germany

    H. F. Fischer

  4. Quantitative Health Sciences, Outcomes Measurement Science, University of Massachusetts Medical School, Worcester, MA, USA

    M. Rose

Authors
  1. S. Nolte
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. Mierke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. H. F. Fischer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. Rose
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. Nolte.

Ethics declarations

Conflict of interest

None.

Ethical approval

Data were acquired as part of routine patient assessment at the Department of Psychosomatic Medicine, Charité—Universitätsmedizin Berlin, Germany. Use of these data for research purposes is covered by §25 of the Regional Hospital Law of Berlin (2011), Landeskrankenhausgesetz (LKG), Berlin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nolte, S., Mierke, A., Fischer, H.F. et al. On the validity of measuring change over time in routine clinical assessment: a close examination of item-level response shifts in psychosomatic inpatients. Qual Life Res 25, 1339–1347 (2016). https://doi.org/10.1007/s11136-015-1123-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-015-1123-3

Keywords

Search

Navigation