Journal of Occupational Rehabilitation

, Volume 24, Issue 4, pp 777–789 | Cite as

Vocational Rehabilitation Program Evaluation: Comparison Group Challenges and the Role of Unmeasured Return-to-Work Expectations

  • Jeanne M. SearsEmail author
  • Lisann R. Rolle
  • Beryl A. Schulman
  • Thomas M. Wickizer


Purpose Despite the importance and cost of workers’ compensation (WC)-based vocational rehabilitation (VR) programs, outcome evaluations are rare, in part due to the scarcity of suitable comparison groups. The aims of this study were to assess (1) the adequacy of a commonly recommended internal comparison group, i.e., workers who were eligible for but did not receive services, and (2) return-to-work (RTW) expectations as a potential source of bias. Methods In this prospective cohort study, we used WC claims data and worker-reported RTW expectations to compare workers who received vocational retraining services to eligible workers who did not receive such services. Workers were surveyed after retraining eligibility determination, prior to the initiation of retraining activities. VR progress and RTW wage outcomes were followed for 3 years. The magnitude of confounding contributed by RTW expectations and other covariates was quantified. Results Workers who were somewhat or very certain they would RTW had significantly better outcomes. RTW expectations played a strong confounding role, reducing the retraining plan effect estimate by about 23 %, while education and physical capacity each changed the effect estimate by <5 %. Conclusions RTW expectations predicted long-term RTW outcomes and can play a strong confounding role if unmeasured. We found that the internal comparison group approach, commonly recommended for VR program evaluation, is inappropriate for WC-based VR evaluations. Ultimately, there is no simple solution to the challenge of identifying a comparison group; however, measurement of RTW expectations, an easily-measured multi-dimensional construct, may be a useful addition to the VR evaluation toolbox.


Vocational rehabilitation Workers’ compensation Expectations Return to work Control groups Confounding factors Program evaluation 



This work was funded by a contract (#K1009) from the Washington State Department of Labor and Industries (L&I). The content of this publication does not necessarily reflect the view or policies of L&I. The authors declare that they have no conflicts of interest. The authors wish to acknowledge Wayne Shatto, Rich Wilson, Kendra Hanson, Sidse Nielsen and many other L&I staff members for their crucial assistance with developing the survey sample, data delivery, and explanations of the very complex data generating processes. Most importantly, we gratefully acknowledge the injured workers who elected to participate in our survey.

Conflict of interest

The authors declare that they have no conflicts of interest.


  1. 1.
    MacEachen E, Kosny A, Ferrier S, Chambers L. The, “toxic dose” of system problems: why some injured workers don’t return to work as expected. J Occup Rehabil. 2010;20(3):349–66.PubMedCrossRefGoogle Scholar
  2. 2.
    Washington State Department of Labor and Industries. Vocational Improvement project report to the legislature: as required by RCW 51.32.099; 2012 December, 2012.Google Scholar
  3. 3.
    Barth P, Grob H, Harder H, Hunt A, Silverstein M. Washington pension system review, Upjohn Institute technical report No. 08-025. Prepared for State of Washington, Department of Labor & Industries, Contract No. K1018. 2008 [cited March 10, 2014].
  4. 4.
    MacEachen E, Kosny A, Ferrier S, Lippel K, Neilson C, Franche RL, et al. The ‘ability’ paradigm in vocational rehabilitation: challenges in an Ontario injured worker retraining program. J Occup Rehabil. 2012;22(1):105–17.PubMedCrossRefGoogle Scholar
  5. 5.
    MacEachen E, Kosny A, Ferrier S, Lippel K, Neilson C, Franche RL, et al. The ideal of consumer choice: challenges with implementation in an Ontario injured worker vocational retraining program. Disabil Rehabil. Online first: 25 April 2013; doi: 10.3109/09638288.2013.771704.
  6. 6.
    Sears JM, Wickizer TM. Evaluation of the vocational rehabilitation pilot program. Report to the Washington State Legislature as required by ESSB 5920 (Chapter 72, Laws of 2007). 2012 [cited March 10, 2014].
  7. 7.
    KPMG LLP. WSIB Labour Market Re-entry (LMR) Program value for money audit report. 2009 [cited March 10, 2014];
  8. 8.
    McPherson K. Evaluation of vocational rehabilitation under the IPRC Act 2001. 2007 [cited March 10, 2014].
  9. 9.
    Bell SH, Orr LL, Blomquist JD, Cain GG. Program applicants as a comparison group in evaluating training programs: theory and a test. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research; 1995.Google Scholar
  10. 10.
    Dean D, Dolan R, Schmidt R, Wehman P, Kregel J, Revell G. A paradigm for evaluation of the federal-state vocational rehabilitation program (Chap. 1). In: Kregel J, Dean D, Wehman P, editors. Achievements and challenges in employment services for people with disabilities: the longitudinal impact of workplace supports. Richmond: Virginia Commonwealth University—Rehabilitation Research and Training Center; 2002.Google Scholar
  11. 11.
    Dean DH. Comparing employment-related outcomes of the vocational rehabilitation program using longitudinal earnings. Am Rehabil. 1991;17(1):4–9.Google Scholar
  12. 12.
    Dean DH, Dolan RC. Assessing the role of vocational rehabilitation in disability policy. J Policy Anal Manage. 1991;10(4):568–87.PubMedCrossRefGoogle Scholar
  13. 13.
    Dean DH, Dolan RC. Fixed-effects estimation of earnings impacts for the vocational rehabilitation program. J Hum Resour. 1991;26(2):380–91.CrossRefGoogle Scholar
  14. 14.
    Dean DH, Dolan RC, Schmidt RM. Evaluating the vocational rehabilitation program using longitudinal data. Evidence for a quasiexperimental research design. Eval Rev. 1999;23(2):162–89.PubMedGoogle Scholar
  15. 15.
    Nowak L. A cost effectiveness evaluation of the federal/state vocational rehabilitation program–using a comparison group. Am Econ. 1983;27(1):23–9.Google Scholar
  16. 16.
    Iles RA, Taylor NF, Davidson M, O’Halloran PD. Patient recovery expectations in non-chronic non-specific low back pain: a qualitative investigation. J Rehabil Med. 2012;44(9):781–7.PubMedCrossRefGoogle Scholar
  17. 17.
    Stewart AM, Polak E, Young R, Schultz IZ. Injured workers’ construction of expectations of return to work with sub-acute back pain: the role of perceived uncertainty. J Occup Rehabil. 2012;22(1):1–14.PubMedCrossRefGoogle Scholar
  18. 18.
    Kapoor S, Shaw WS, Pransky G, Patterson W. Initial patient and clinician expectations of return to work after acute onset of work-related low back pain. J Occup Environ Med. 2006;48(11):1173–80.PubMedCrossRefGoogle Scholar
  19. 19.
    Lovvik C, Overland S, Hysing M, Broadbent E, Reme SE. Association between illness perceptions and return-to-work expectations in workers with common mental health symptoms. J Occup Rehabil. 2013.Google Scholar
  20. 20.
    Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191–215.PubMedCrossRefGoogle Scholar
  21. 21.
    Cole DC, Mondloch MV, Hogg-Johnson S. Listening to injured workers: how recovery expectations predict outcomes—a prospective study. CMAJ. 2002;166(6):749–54.PubMedCentralPubMedGoogle Scholar
  22. 22.
    Gross DP, Battie MC. Work-related recovery expectations and the prognosis of chronic low back pain within a workers’ compensation setting. J Occup Environ Med. 2005;47(4):428–33.PubMedCrossRefGoogle Scholar
  23. 23.
    Gross DP, Battie MC. Recovery expectations predict recovery in workers with back pain but not other musculoskeletal conditions. J Spinal Disord Tech. 2010;23(7):451–6.PubMedCrossRefGoogle Scholar
  24. 24.
    Heijbel B, Josephson M, Jensen I, Stark S, Vingard E. Return to work expectation predicts work in chronic musculoskeletal and behavioral health disorders: prospective study with clinical implications. J Occup Rehabil. 2006;16(2):173–84.PubMedCrossRefGoogle Scholar
  25. 25.
    Hogg-Johnson S, Cole DC. Early prognostic factors for duration on temporary total benefits in the first year among workers with compensated occupational soft tissue injuries. Occup Environ Med. 2003;60(4):244–53.PubMedCentralPubMedCrossRefGoogle Scholar
  26. 26.
    Laisne F, Lecomte C, Corbiere M. Biopsychosocial determinants of work outcomes of workers with occupational injuries receiving compensation: a prospective study. Work. 2013;44(2):117–32.PubMedGoogle Scholar
  27. 27.
    Turner JA, Franklin G, Fulton-Kehoe D, Sheppard L, Wickizer TM, Wu R, et al. Worker recovery expectations and fear-avoidance predict work disability in a population-based workers’ compensation back pain sample. Spine (Phila Pa 1976). 2006;31(6):682–9.CrossRefGoogle Scholar
  28. 28.
    Fadyl J, McPherson K. Return to work after injury: a review of evidence regarding expectations and injury perceptions, and their influence on outcome. J Occup Rehabil. 2008;18(4):362–74.PubMedCrossRefGoogle Scholar
  29. 29.
    Gray H, Adefolarin AT, Howe TE. A systematic review of instruments for the assessment of work-related psychosocial factors (Blue Flags) in individuals with non-specific low back pain. Man Ther. 2011;16(6):531–43.PubMedCrossRefGoogle Scholar
  30. 30.
    Hallegraeff JM, Krijnen WP, van der Schans CP, de Greef MH. Expectations about recovery from acute non-specific low back pain predict absence from usual work due to chronic low back pain: a systematic review. J Physiother. 2012;58(3):165–72.PubMedCrossRefGoogle Scholar
  31. 31.
    Iles RA, Davidson M, Taylor NF. Psychosocial predictors of failure to return to work in non-chronic non-specific low back pain: a systematic review. Occup Environ Med. 2008;65(8):507–17.PubMedCrossRefGoogle Scholar
  32. 32.
    Iles RA, Davidson M, Taylor NF, O’Halloran P. Systematic review of the ability of recovery expectations to predict outcomes in non-chronic non-specific low back pain. J Occup Rehabil. 2009;19(1):25–40.PubMedCrossRefGoogle Scholar
  33. 33.
    State of Washington. RCW Title 51: Chapter 51.12. Employments and occupations covered. [cited March 10, 2014].
  34. 34.
    Tanabe RP. Workers’ compensation laws as of January 2012. WC-12-18. Cambridge, Mass.: Workers Compensation Research Institute; 2012 March 2012.Google Scholar
  35. 35.
    Franklin GM, Fulton-Kehoe D. Outcomes research in Washington state workers’ compensation. Am J Ind Med. 1996;29(6):642–8.PubMedCrossRefGoogle Scholar
  36. 36.
    Franklin GM, Wickizer TM, Fulton-Kehoe D, Turner JA. Policy-relevant research: when does it matter? NeuroRx. 2004;1(3):356–62.PubMedCentralPubMedCrossRefGoogle Scholar
  37. 37.
    Washington State Department of Labor and Industries. Workers’ compensation benefits: a guide for injured workers. Publication F242-104-000 [03-2012]. 2012 [cited March 10, 2014].
  38. 38.
    The American Association for Public Opinion Research. Standard definitions: final dispositions of case codes and outcome rates for surveys. 5th ed. Lenexa, KS: AAPOR; 2008.Google Scholar
  39. 39.
    Sears JM, Wickizer TM, Schulman BA. Injured workers’ assessment of vocational rehabilitation services before and after retraining. J Occup Rehabil. Online first: 25 Sept 2013. doi: 10.1007/s10926-013-9479-0.
  40. 40.
    Washington State Employment Security Department. Quarterly census of employment and wages, fourth quarter 2011 preliminary. LM-12-0263.; July 2012.Google Scholar
  41. 41.
    Sears JM, Wickizer TM, Schulman BA. Improving vocational rehabilitation services for injured workers in Washington State. Eval Program Plann. 2014;44:26–35.PubMedCrossRefGoogle Scholar
  42. 42.
    Wang Z. Two postestimation commands for assessing confounding effects in epidemiological studies. Stata J. 2007;7(2):183–96.Google Scholar
  43. 43.
    Simon JG, De Boer JB, Joung IM, Bosma H, Mackenbach JP. How is your health in general? A qualitative study on self-assessed health. Eur J Public Health. 2005;15(2):200–8.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Jeanne M. Sears
    • 1
    Email author
  • Lisann R. Rolle
    • 2
  • Beryl A. Schulman
    • 1
  • Thomas M. Wickizer
    • 3
  1. 1.Department of Health Services, School of Public HealthUniversity of WashingtonSeattleUSA
  2. 2.Washington State Department of Labor and IndustriesTumwaterUSA
  3. 3.Division of Health Services Management and PolicyThe Ohio State University College of Public HealthColumbusUSA

Personalised recommendations