Skip to main content
SpringerLink
Log in

The inclusion of ‘then-test’ questions in post-test questionnaires alters post-test responses: a randomized study of bias in health program evaluation

  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Objectives

Program evaluations are frequently based on ‘then-test’ data, i.e., pre-test collected in retrospect. While the application of the then-test has practical advantages, little is known about the validity of then-test data. Because of the collection of then-test in close proximity to post-test questions, this study was aimed at exploring whether the presence of then-test questions in post-test questionnaires influenced subjects’ responses to post-test.

Patients and methods

To test the influence of then-test questions, we designed a randomized three-group study in the context of chronic disease self-management programs. Interventions had comparable goals and philosophies, and all 949 study participants filled out identical Health Education Impact Questionnaires (heiQ) at pre-test. At post-test, participants were then randomized to one of the following three groups: Group A responded to post-test questions only (n = 331); Group B filled out transition questions in addition to post-test (n = 304); and Group C filled out then-test questions in addition to post-test (n = 314).

Results

Significant post-test differences were found in six of eight heiQ scales, with respondents who filled out then-test questions reporting significantly higher post-test scores than respondents of the other groups.

Conclusions

This study provides evidence that the inclusion of then-test questions alters post-test responses, suggesting that change scores based on then-test data be interpreted with care.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Golembiewsky, R., Billingsley, K., & Yeager, S. (1976). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavioral Science, 12, 133–157.

    Article  Google Scholar 

  2. Howard, G. S., & Dailey, P. R. (1979). Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64(2), 144–150.

    Article  Google Scholar 

  3. Sprangers, M. A. G., & Schwartz, C. E. (1999). Integrating response shift into health-related quality of life research: A theoretical model. Social Science and Medicine, 48(11), 1507–1515.

    Article  PubMed  CAS  Google Scholar 

  4. Schwartz, C. E., & Sprangers, M. A. G. (1999). Methodological approaches for assessing response shift in longitudinal health-related quality-of-life research. Social Science and Medicine, 48(11), 1531–1548.

    Article  PubMed  CAS  Google Scholar 

  5. Cronbach, L. J., & Furby, L. (1970). How should we measure “change”—or should we? Psychological Bulletin, 74(1), 68–80.

    Article  Google Scholar 

  6. Lorig, K. R., Sobel, D. S., Stewart, A. L., Brown, B. W., Bandura, A., Ritter, P., et al. (1999). Evidence suggesting that a chronic disease self-management program can improve health status while reducing hospitalization. Medical Care, 37(1), 5–14.

    Article  PubMed  CAS  Google Scholar 

  7. Elzen, H., Slaets, J., Snijders, T., & Steverink, N. (2007). Evaluation of the chronic disease self-management program (CDSMP) among chronically ill older people in the Netherlands. Social Science and Medicine, 64, 1832–1841.

    Article  PubMed  Google Scholar 

  8. Lorig, K. R. (2003). Editoria—Self-management education: more than a nice extra. Medical Care, 41(6), 699–701.

    PubMed  Google Scholar 

  9. Lorig, K. R., & Holman, H. R. (1989). Long-term outcomes of an arthritis self-management study: Effects of reinforcement efforts. Social Science and Medicine, 29(2), 221–224.

    Article  PubMed  CAS  Google Scholar 

  10. Gibbons, F. X. (1999). Social comparison as a mediator of response shift. Social Science and Medicine, 48(11), 1517–1530.

    Article  PubMed  CAS  Google Scholar 

  11. Osborne, R. H., Hawkins, M., & Sprangers, M. A. G. (2006). Change of perspective: A measurable and desired outcome of chronic disease self-management intervention programs that violates the premise of preintervention/postintervention assessment. Arthritis Care and Research, 55(3), 458–465.

    Article  PubMed  Google Scholar 

  12. Sprangers, M. A. G. (1989). Response-shift bias in program evaluation. Impact Assessment Bulletin, 7, 153–166.

    Google Scholar 

  13. Schwartz, C., & Sprangers, M. (2010). Guidelines for improving the stringency of response shift research using the thentest. Quality of Life Research, 19, 455–464.

    Article  PubMed  Google Scholar 

  14. Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105, 456–466.

    Article  Google Scholar 

  15. McGaw, B., & Jöreskog, K. G. (1971). Factorial invariance of ability measures in groups differing in intelligence and socioeconomic status. British Journal of Mathematical, 24, 154–168.

    Google Scholar 

  16. Nolte, S., Elsworth, G. R., Sinclair, A. J., & Osborne, R. H. (2009). Tests of measurement invariance failed to support the application of the “then-test”. Journal of Clinical Epidemiology, 62(11), 1173–1180.

    Article  PubMed  Google Scholar 

  17. Howard, G. S., Schmeck, R. R., & Bray, J. H. (1979). Internal invalidity in studies employing self-report instruments: a suggested remedy. Journal of Educational Measurement, 16(2), 129–135.

    Article  Google Scholar 

  18. Lorig, K., & Fries, J. (2000). The arthritis helpbook (5th ed.). Cambridge, MA: Perseus Books.

    Google Scholar 

  19. Lorig, K. R., González, V. M., & Laurent, D. D. (1999). The chronic disease self-management program: Leaders manual. Palo Alto: Stanford University.

    Google Scholar 

  20. Osteoporosis Victoria. (2001). The osteoporosis prevention and self management course—Leaders manual. Elsternwick: Osteoporosis Victoria, A division of Arthritis Victoria.

    Google Scholar 

  21. Nolte, S., Elsworth, G. R., Sinclair, A. J., & Osborne, R. H. (2007). The extent and breadth of benefits from participating in chronic disease self-management courses: A national patient-reported outcomes survey. Patient Education and Counseling, 65(3), 351–360.

    Article  PubMed  Google Scholar 

  22. Osborne, R. H., Elsworth, G. R., & Whitfield, K. (2007). The health education impact questionnaire (heiQ): An outcomes and evaluation measure for patient education and self-management interventions for people with chronic conditions. Patient Education and Counseling, 66, 192–201.

    Article  PubMed  Google Scholar 

  23. Nolte, S. (2008). Approaches to the measurement of outcomes of chronic disease self-management interventions using a self-report inventory. Melbourne: RMIT University.

    Google Scholar 

  24. Brown, M. B., & Forsythe, A. B. (1974). Robust tests for the equality of variances. Journal of the American Statistical Association, 69(346), 364–367.

    Article  Google Scholar 

  25. Rubin, D. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.

    Article  Google Scholar 

  26. Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567.

    Article  PubMed  CAS  Google Scholar 

  27. Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51, 201–219.

    Article  Google Scholar 

  28. Lam, T. C. M., & Bengo, P. (2003). A comparison of three retrospective self-reporting methods of measuring change in instructional practice. American Journal of Evaluation, 24(1), 65–80.

    Google Scholar 

  29. Schwarz, N., & Strack, F. (1985). Cognitive and affective processes in judgements of subjective well-being: a preliminary model. In H. Brandstatter & E. Kirchler (Eds.), Economic psychology (pp. 439–447). Linz, Austria: R. Tauner.

    Google Scholar 

  30. Crowne, D. P., & Marlowe, D. (1964). The approval motive: Studies in evaluative dependence. New York, London, Sydney: Wiley.

    Google Scholar 

  31. Howard, G. S., Ralph, K. M., Gulanick, N. A., Maxwell, S. E., Nance, S. W., & Gerber, S. K. (1979). Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3, 1–23.

    Article  Google Scholar 

  32. Hill, L. G., & Betz, D. L. (2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26(4), 501–517.

    Article  Google Scholar 

  33. Sprangers, M. A. G., & Hoogstraten, J. (1988). On delay and reassessment of retrospective preratings. Journal of Experimental Education, 56, 148–153.

    Google Scholar 

  34. Norman, G. (2003). Hi! How are you? Response shift, implicit theories and differing epistemologies. Quality of Life Research, 12, 239–249.

    Article  PubMed  Google Scholar 

  35. Ross, M. (1989). Relation of implicit theories to the construction of personal histories. Psychological Review, 96, 341–357.

    Article  Google Scholar 

  36. Nimon, K., Zigarmi, D., & Allen, J. (2011). Measures of program effectiveness based on retrospective pretest data: are all created equal? American Journal of Evaluation, 32(8), 8–28.

    Article  Google Scholar 

  37. Terborg, J., & Davis, G. (1982). Evaluation of a new method of assessing change to planned job redesign as applied to Hackman and Oldham’s job characteristic model. Organizational Behavior and Human Performance, 29, 112–128.

    Article  Google Scholar 

  38. Sprangers, M., & Hoogstraten, J. (1989). Pretesting effects in retrospective pretest-posttest designs. Journal of Applied Psychology, 74, 265–272.

    Article  Google Scholar 

  39. Stone, G. R., & Packer, T. L. (2010). Evaluation of a rural chronic disease self-management program. Rural and Remote Health, 10(1), 1203.

    PubMed  Google Scholar 

Download references

Acknowledgments

The authors thank Mary Ljubanovic, Amanda Springer, and Stella Vo for administrative support, the Australian Arthritis Foundations for support with recruitment of course leaders, all participants who kindly took part in the study, and the many coordinators and course leaders who administered the questionnaires.

Author information

Authors and Affiliations

  1. Deakin University, Burwood, VIC, Australia

    Sandra Nolte, Gerald R. Elsworth, Andrew J. Sinclair & Richard H. Osborne

Authors
  1. Sandra Nolte
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gerald R. Elsworth
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrew J. Sinclair
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard H. Osborne
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sandra Nolte.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nolte, S., Elsworth, G.R., Sinclair, A.J. et al. The inclusion of ‘then-test’ questions in post-test questionnaires alters post-test responses: a randomized study of bias in health program evaluation. Qual Life Res 21, 487–494 (2012). https://doi.org/10.1007/s11136-011-9952-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-011-9952-1

Keywords

Search

Navigation