Vocational Rehabilitation Program Evaluation: Comparison Group Challenges and the Role of Unmeasured Return-to-Work Expectations

Abstract

Purpose Despite the importance and cost of workers’ compensation (WC)-based vocational rehabilitation (VR) programs, outcome evaluations are rare, in part due to the scarcity of suitable comparison groups. The aims of this study were to assess (1) the adequacy of a commonly recommended internal comparison group, i.e., workers who were eligible for but did not receive services, and (2) return-to-work (RTW) expectations as a potential source of bias. Methods In this prospective cohort study, we used WC claims data and worker-reported RTW expectations to compare workers who received vocational retraining services to eligible workers who did not receive such services. Workers were surveyed after retraining eligibility determination, prior to the initiation of retraining activities. VR progress and RTW wage outcomes were followed for 3 years. The magnitude of confounding contributed by RTW expectations and other covariates was quantified. Results Workers who were somewhat or very certain they would RTW had significantly better outcomes. RTW expectations played a strong confounding role, reducing the retraining plan effect estimate by about 23 %, while education and physical capacity each changed the effect estimate by <5 %. Conclusions RTW expectations predicted long-term RTW outcomes and can play a strong confounding role if unmeasured. We found that the internal comparison group approach, commonly recommended for VR program evaluation, is inappropriate for WC-based VR evaluations. Ultimately, there is no simple solution to the challenge of identifying a comparison group; however, measurement of RTW expectations, an easily-measured multi-dimensional construct, may be a useful addition to the VR evaluation toolbox.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3

Notes

  1. 1.

    Data for this methodological study were drawn from the Abt Associates’ evaluation of the Aid to Families with Dependent Children (AFDC) Homemaker-Home Health Aide Demonstrations.

References

  1. 1.

    MacEachen E, Kosny A, Ferrier S, Chambers L. The, “toxic dose” of system problems: why some injured workers don’t return to work as expected. J Occup Rehabil. 2010;20(3):349–66.

    PubMed  Article  Google Scholar 

  2. 2.

    Washington State Department of Labor and Industries. Vocational Improvement project report to the legislature: as required by RCW 51.32.099; 2012 December, 2012.

  3. 3.

    Barth P, Grob H, Harder H, Hunt A, Silverstein M. Washington pension system review, Upjohn Institute technical report No. 08-025. Prepared for State of Washington, Department of Labor & Industries, Contract No. K1018. 2008 [cited March 10, 2014]. http://www.upjohninst.org/publications/tr/tr08-025.pdf.

  4. 4.

    MacEachen E, Kosny A, Ferrier S, Lippel K, Neilson C, Franche RL, et al. The ‘ability’ paradigm in vocational rehabilitation: challenges in an Ontario injured worker retraining program. J Occup Rehabil. 2012;22(1):105–17.

    CAS  PubMed  Article  Google Scholar 

  5. 5.

    MacEachen E, Kosny A, Ferrier S, Lippel K, Neilson C, Franche RL, et al. The ideal of consumer choice: challenges with implementation in an Ontario injured worker vocational retraining program. Disabil Rehabil. Online first: 25 April 2013; doi:10.3109/09638288.2013.771704.

  6. 6.

    Sears JM, Wickizer TM. Evaluation of the vocational rehabilitation pilot program. Report to the Washington State Legislature as required by ESSB 5920 (Chapter 72, Laws of 2007). 2012 [cited March 10, 2014]. http://www.lni.wa.gov/ClaimsIns/Files/Vocational/VocPilotProgEval.pdf.

  7. 7.

    KPMG LLP. WSIB Labour Market Re-entry (LMR) Program value for money audit report. 2009 [cited March 10, 2014]; http://www.wsib.on.ca/files/Content/VFMAVFMALMR2009/LMRvfma.pdf.

  8. 8.

    McPherson K. Evaluation of vocational rehabilitation under the IPRC Act 2001. 2007 [cited March 10, 2014]. http://hazelarmstronglaw.co.nz/beta/wp-content/uploads/2012/07/Evaluation-of-Vocational-Rehabilitation-under-the-IPRC-Act-2001-Kathryn-McPherson.pdf.

  9. 9.

    Bell SH, Orr LL, Blomquist JD, Cain GG. Program applicants as a comparison group in evaluating training programs: theory and a test. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research; 1995.

    Google Scholar 

  10. 10.

    Dean D, Dolan R, Schmidt R, Wehman P, Kregel J, Revell G. A paradigm for evaluation of the federal-state vocational rehabilitation program (Chap. 1). In: Kregel J, Dean D, Wehman P, editors. Achievements and challenges in employment services for people with disabilities: the longitudinal impact of workplace supports. Richmond: Virginia Commonwealth University—Rehabilitation Research and Training Center; 2002.

    Google Scholar 

  11. 11.

    Dean DH. Comparing employment-related outcomes of the vocational rehabilitation program using longitudinal earnings. Am Rehabil. 1991;17(1):4–9.

    Google Scholar 

  12. 12.

    Dean DH, Dolan RC. Assessing the role of vocational rehabilitation in disability policy. J Policy Anal Manage. 1991;10(4):568–87.

    CAS  PubMed  Article  Google Scholar 

  13. 13.

    Dean DH, Dolan RC. Fixed-effects estimation of earnings impacts for the vocational rehabilitation program. J Hum Resour. 1991;26(2):380–91.

    Article  Google Scholar 

  14. 14.

    Dean DH, Dolan RC, Schmidt RM. Evaluating the vocational rehabilitation program using longitudinal data. Evidence for a quasiexperimental research design. Eval Rev. 1999;23(2):162–89.

    CAS  PubMed  Google Scholar 

  15. 15.

    Nowak L. A cost effectiveness evaluation of the federal/state vocational rehabilitation program–using a comparison group. Am Econ. 1983;27(1):23–9.

    Google Scholar 

  16. 16.

    Iles RA, Taylor NF, Davidson M, O’Halloran PD. Patient recovery expectations in non-chronic non-specific low back pain: a qualitative investigation. J Rehabil Med. 2012;44(9):781–7.

    PubMed  Article  Google Scholar 

  17. 17.

    Stewart AM, Polak E, Young R, Schultz IZ. Injured workers’ construction of expectations of return to work with sub-acute back pain: the role of perceived uncertainty. J Occup Rehabil. 2012;22(1):1–14.

    PubMed  Article  Google Scholar 

  18. 18.

    Kapoor S, Shaw WS, Pransky G, Patterson W. Initial patient and clinician expectations of return to work after acute onset of work-related low back pain. J Occup Environ Med. 2006;48(11):1173–80.

    PubMed  Article  Google Scholar 

  19. 19.

    Lovvik C, Overland S, Hysing M, Broadbent E, Reme SE. Association between illness perceptions and return-to-work expectations in workers with common mental health symptoms. J Occup Rehabil. 2013.

  20. 20.

    Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191–215.

    CAS  PubMed  Article  Google Scholar 

  21. 21.

    Cole DC, Mondloch MV, Hogg-Johnson S. Listening to injured workers: how recovery expectations predict outcomes—a prospective study. CMAJ. 2002;166(6):749–54.

    PubMed Central  PubMed  Google Scholar 

  22. 22.

    Gross DP, Battie MC. Work-related recovery expectations and the prognosis of chronic low back pain within a workers’ compensation setting. J Occup Environ Med. 2005;47(4):428–33.

    PubMed  Article  Google Scholar 

  23. 23.

    Gross DP, Battie MC. Recovery expectations predict recovery in workers with back pain but not other musculoskeletal conditions. J Spinal Disord Tech. 2010;23(7):451–6.

    PubMed  Article  Google Scholar 

  24. 24.

    Heijbel B, Josephson M, Jensen I, Stark S, Vingard E. Return to work expectation predicts work in chronic musculoskeletal and behavioral health disorders: prospective study with clinical implications. J Occup Rehabil. 2006;16(2):173–84.

    PubMed  Article  Google Scholar 

  25. 25.

    Hogg-Johnson S, Cole DC. Early prognostic factors for duration on temporary total benefits in the first year among workers with compensated occupational soft tissue injuries. Occup Environ Med. 2003;60(4):244–53.

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  26. 26.

    Laisne F, Lecomte C, Corbiere M. Biopsychosocial determinants of work outcomes of workers with occupational injuries receiving compensation: a prospective study. Work. 2013;44(2):117–32.

    PubMed  Google Scholar 

  27. 27.

    Turner JA, Franklin G, Fulton-Kehoe D, Sheppard L, Wickizer TM, Wu R, et al. Worker recovery expectations and fear-avoidance predict work disability in a population-based workers’ compensation back pain sample. Spine (Phila Pa 1976). 2006;31(6):682–9.

    Article  Google Scholar 

  28. 28.

    Fadyl J, McPherson K. Return to work after injury: a review of evidence regarding expectations and injury perceptions, and their influence on outcome. J Occup Rehabil. 2008;18(4):362–74.

    PubMed  Article  Google Scholar 

  29. 29.

    Gray H, Adefolarin AT, Howe TE. A systematic review of instruments for the assessment of work-related psychosocial factors (Blue Flags) in individuals with non-specific low back pain. Man Ther. 2011;16(6):531–43.

    PubMed  Article  Google Scholar 

  30. 30.

    Hallegraeff JM, Krijnen WP, van der Schans CP, de Greef MH. Expectations about recovery from acute non-specific low back pain predict absence from usual work due to chronic low back pain: a systematic review. J Physiother. 2012;58(3):165–72.

    PubMed  Article  Google Scholar 

  31. 31.

    Iles RA, Davidson M, Taylor NF. Psychosocial predictors of failure to return to work in non-chronic non-specific low back pain: a systematic review. Occup Environ Med. 2008;65(8):507–17.

    CAS  PubMed  Article  Google Scholar 

  32. 32.

    Iles RA, Davidson M, Taylor NF, O’Halloran P. Systematic review of the ability of recovery expectations to predict outcomes in non-chronic non-specific low back pain. J Occup Rehabil. 2009;19(1):25–40.

    PubMed  Article  Google Scholar 

  33. 33.

    State of Washington. RCW Title 51: Chapter 51.12. Employments and occupations covered. [cited March 10, 2014]. http://apps.leg.wa.gov/rcw/default.aspx?Cite=51.12.

  34. 34.

    Tanabe RP. Workers’ compensation laws as of January 2012. WC-12-18. Cambridge, Mass.: Workers Compensation Research Institute; 2012 March 2012.

  35. 35.

    Franklin GM, Fulton-Kehoe D. Outcomes research in Washington state workers’ compensation. Am J Ind Med. 1996;29(6):642–8.

    CAS  PubMed  Article  Google Scholar 

  36. 36.

    Franklin GM, Wickizer TM, Fulton-Kehoe D, Turner JA. Policy-relevant research: when does it matter? NeuroRx. 2004;1(3):356–62.

    PubMed Central  PubMed  Article  Google Scholar 

  37. 37.

    Washington State Department of Labor and Industries. Workers’ compensation benefits: a guide for injured workers. Publication F242-104-000 [03-2012]. 2012 [cited March 10, 2014]. http://www.lni.wa.gov/IPUB/242-104-000.pdf.

  38. 38.

    The American Association for Public Opinion Research. Standard definitions: final dispositions of case codes and outcome rates for surveys. 5th ed. Lenexa, KS: AAPOR; 2008.

    Google Scholar 

  39. 39.

    Sears JM, Wickizer TM, Schulman BA. Injured workers’ assessment of vocational rehabilitation services before and after retraining. J Occup Rehabil. Online first: 25 Sept 2013. doi:10.1007/s10926-013-9479-0.

  40. 40.

    Washington State Employment Security Department. Quarterly census of employment and wages, fourth quarter 2011 preliminary. LM-12-0263.; July 2012.

  41. 41.

    Sears JM, Wickizer TM, Schulman BA. Improving vocational rehabilitation services for injured workers in Washington State. Eval Program Plann. 2014;44:26–35.

    PubMed  Article  Google Scholar 

  42. 42.

    Wang Z. Two postestimation commands for assessing confounding effects in epidemiological studies. Stata J. 2007;7(2):183–96.

    Google Scholar 

  43. 43.

    Simon JG, De Boer JB, Joung IM, Bosma H, Mackenbach JP. How is your health in general? A qualitative study on self-assessed health. Eur J Public Health. 2005;15(2):200–8.

    CAS  PubMed  Article  Google Scholar 

Download references

Acknowledgments

This work was funded by a contract (#K1009) from the Washington State Department of Labor and Industries (L&I). The content of this publication does not necessarily reflect the view or policies of L&I. The authors declare that they have no conflicts of interest. The authors wish to acknowledge Wayne Shatto, Rich Wilson, Kendra Hanson, Sidse Nielsen and many other L&I staff members for their crucial assistance with developing the survey sample, data delivery, and explanations of the very complex data generating processes. Most importantly, we gratefully acknowledge the injured workers who elected to participate in our survey.

Conflict of interest

The authors declare that they have no conflicts of interest.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jeanne M. Sears.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Sears, J.M., Rolle, L.R., Schulman, B.A. et al. Vocational Rehabilitation Program Evaluation: Comparison Group Challenges and the Role of Unmeasured Return-to-Work Expectations. J Occup Rehabil 24, 777–789 (2014). https://doi.org/10.1007/s10926-014-9509-6

Download citation

Keywords

  • Vocational rehabilitation
  • Workers’ compensation
  • Expectations
  • Return to work
  • Control groups
  • Confounding factors
  • Program evaluation