Vocational Rehabilitation Program Evaluation: Comparison Group Challenges and the Role of Unmeasured Return-to-Work Expectations
- 453 Downloads
Purpose Despite the importance and cost of workers’ compensation (WC)-based vocational rehabilitation (VR) programs, outcome evaluations are rare, in part due to the scarcity of suitable comparison groups. The aims of this study were to assess (1) the adequacy of a commonly recommended internal comparison group, i.e., workers who were eligible for but did not receive services, and (2) return-to-work (RTW) expectations as a potential source of bias. Methods In this prospective cohort study, we used WC claims data and worker-reported RTW expectations to compare workers who received vocational retraining services to eligible workers who did not receive such services. Workers were surveyed after retraining eligibility determination, prior to the initiation of retraining activities. VR progress and RTW wage outcomes were followed for 3 years. The magnitude of confounding contributed by RTW expectations and other covariates was quantified. Results Workers who were somewhat or very certain they would RTW had significantly better outcomes. RTW expectations played a strong confounding role, reducing the retraining plan effect estimate by about 23 %, while education and physical capacity each changed the effect estimate by <5 %. Conclusions RTW expectations predicted long-term RTW outcomes and can play a strong confounding role if unmeasured. We found that the internal comparison group approach, commonly recommended for VR program evaluation, is inappropriate for WC-based VR evaluations. Ultimately, there is no simple solution to the challenge of identifying a comparison group; however, measurement of RTW expectations, an easily-measured multi-dimensional construct, may be a useful addition to the VR evaluation toolbox.
KeywordsVocational rehabilitation Workers’ compensation Expectations Return to work Control groups Confounding factors Program evaluation
This work was funded by a contract (#K1009) from the Washington State Department of Labor and Industries (L&I). The content of this publication does not necessarily reflect the view or policies of L&I. The authors declare that they have no conflicts of interest. The authors wish to acknowledge Wayne Shatto, Rich Wilson, Kendra Hanson, Sidse Nielsen and many other L&I staff members for their crucial assistance with developing the survey sample, data delivery, and explanations of the very complex data generating processes. Most importantly, we gratefully acknowledge the injured workers who elected to participate in our survey.
Conflict of interest
The authors declare that they have no conflicts of interest.
- 2.Washington State Department of Labor and Industries. Vocational Improvement project report to the legislature: as required by RCW 51.32.099; 2012 December, 2012.Google Scholar
- 3.Barth P, Grob H, Harder H, Hunt A, Silverstein M. Washington pension system review, Upjohn Institute technical report No. 08-025. Prepared for State of Washington, Department of Labor & Industries, Contract No. K1018. 2008 [cited March 10, 2014]. http://www.upjohninst.org/publications/tr/tr08-025.pdf.
- 5.MacEachen E, Kosny A, Ferrier S, Lippel K, Neilson C, Franche RL, et al. The ideal of consumer choice: challenges with implementation in an Ontario injured worker vocational retraining program. Disabil Rehabil. Online first: 25 April 2013; doi: 10.3109/09638288.2013.771704.
- 6.Sears JM, Wickizer TM. Evaluation of the vocational rehabilitation pilot program. Report to the Washington State Legislature as required by ESSB 5920 (Chapter 72, Laws of 2007). 2012 [cited March 10, 2014]. http://www.lni.wa.gov/ClaimsIns/Files/Vocational/VocPilotProgEval.pdf.
- 7.KPMG LLP. WSIB Labour Market Re-entry (LMR) Program value for money audit report. 2009 [cited March 10, 2014]; http://www.wsib.on.ca/files/Content/VFMAVFMALMR2009/LMRvfma.pdf.
- 8.McPherson K. Evaluation of vocational rehabilitation under the IPRC Act 2001. 2007 [cited March 10, 2014]. http://hazelarmstronglaw.co.nz/beta/wp-content/uploads/2012/07/Evaluation-of-Vocational-Rehabilitation-under-the-IPRC-Act-2001-Kathryn-McPherson.pdf.
- 9.Bell SH, Orr LL, Blomquist JD, Cain GG. Program applicants as a comparison group in evaluating training programs: theory and a test. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research; 1995.Google Scholar
- 10.Dean D, Dolan R, Schmidt R, Wehman P, Kregel J, Revell G. A paradigm for evaluation of the federal-state vocational rehabilitation program (Chap. 1). In: Kregel J, Dean D, Wehman P, editors. Achievements and challenges in employment services for people with disabilities: the longitudinal impact of workplace supports. Richmond: Virginia Commonwealth University—Rehabilitation Research and Training Center; 2002.Google Scholar
- 11.Dean DH. Comparing employment-related outcomes of the vocational rehabilitation program using longitudinal earnings. Am Rehabil. 1991;17(1):4–9.Google Scholar
- 15.Nowak L. A cost effectiveness evaluation of the federal/state vocational rehabilitation program–using a comparison group. Am Econ. 1983;27(1):23–9.Google Scholar
- 19.Lovvik C, Overland S, Hysing M, Broadbent E, Reme SE. Association between illness perceptions and return-to-work expectations in workers with common mental health symptoms. J Occup Rehabil. 2013.Google Scholar
- 33.State of Washington. RCW Title 51: Chapter 51.12. Employments and occupations covered. [cited March 10, 2014]. http://apps.leg.wa.gov/rcw/default.aspx?Cite=51.12.
- 34.Tanabe RP. Workers’ compensation laws as of January 2012. WC-12-18. Cambridge, Mass.: Workers Compensation Research Institute; 2012 March 2012.Google Scholar
- 37.Washington State Department of Labor and Industries. Workers’ compensation benefits: a guide for injured workers. Publication F242-104-000 [03-2012]. 2012 [cited March 10, 2014]. http://www.lni.wa.gov/IPUB/242-104-000.pdf.
- 38.The American Association for Public Opinion Research. Standard definitions: final dispositions of case codes and outcome rates for surveys. 5th ed. Lenexa, KS: AAPOR; 2008.Google Scholar
- 39.Sears JM, Wickizer TM, Schulman BA. Injured workers’ assessment of vocational rehabilitation services before and after retraining. J Occup Rehabil. Online first: 25 Sept 2013. doi: 10.1007/s10926-013-9479-0.
- 40.Washington State Employment Security Department. Quarterly census of employment and wages, fourth quarter 2011 preliminary. LM-12-0263.; July 2012.Google Scholar
- 42.Wang Z. Two postestimation commands for assessing confounding effects in epidemiological studies. Stata J. 2007;7(2):183–96.Google Scholar