Improving the measurement accuracy of the effort-reward imbalance scales

  • Akizumi Tsutsumi
  • Noboru Iwata
  • Takafumi Wakita
  • Ryuichi Kumagai
  • Hiroyuki Noguchi
  • Norito Kawakami


Background: The effort-reward imbalance (ERI) scale items are answered in a two-step process, but the justification is questioned for the formulation of summary measure by combining information rated in two steps. Purpose: To examine whether the basic prerequisites of the ERI scales are empirically satisfied and to seek ways to improve the rating procedure. Methods: A polytomous item response theory (IRT) model was applied to the responses of 20,256 workers who completed the ERI scales. To determine the most appropriate statistical justification, three alternative scoring algorithms were compared with regard to the test properties revealed by the IRT analyses and efficiencies of screening performance and criterion validity against depressive symptomatology. Results: The rated raw-score units did not reflect the hypothesized order of lowest stress levels to highest stress levels. Exchanging or collapsing the lowest two categories of a Likert scaled item, where data of different quality are combined, solved this problem, thereby making the test content more appropriate. The modified rating improved the efficiencies of screening performance and the correlation of the stress summary measures against health criterion, i.e., depression. Conclusion: An avoidable measurement error exists in the current ERI scales. Modifying the rating procedure can improve the measurement accuracy.

Key words

effort-reward imbalance item response theory measurement accuracy psychological testing questionnaire stress 


  1. Adams, R. J. (1988). Applying the partial credit model to educational diagnosis. Applied Measurement in Education, 1(4), 347–361.CrossRefGoogle Scholar
  2. Aggen, S. H., Neale, M. C., & Kendler, K. S. (2005). DSM criteria for major depression: Evaluating symptom patterns using latent-trait item response models. Psychological Medicine, 35(4), 475–487.PubMedCrossRefGoogle Scholar
  3. Andrich, D. (1988). A general form of Rasch’s extended logistic model for partial credit. Applied Measurement in Education, 1(4), 363–378.CrossRefGoogle Scholar
  4. Cook, C. E., Richardson, J. K., & Pietrobon, R. (2006). Examination surveys activities of daily living instrument among patients with report of low back pain. Journal of Manipulative and Physiological Therapeutics,29(3), 183–189.PubMedCrossRefGoogle Scholar
  5. Cook, K., Ashton, C., Byrne, M., Brody, B., Geraci, J., Giesler, R., et al. (2001). A psychometric analysis of the measurement level of the rating scale, time trade-off, and standard gamble. Social Science & Medicine, 53, 1275–1285.CrossRefGoogle Scholar
  6. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  7. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston, MA: Kluwer.Google Scholar
  8. Hambleton, R.K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.Google Scholar
  9. Hattie, J. (1984). An empirical study of various indices for determining unidimensionality. Multivariate Behavioral Research, 19, 49–78.CrossRefGoogle Scholar
  10. Head, J., Stansfeld, S. A., & Siegrist, J. (2004). The psychosocial work environment and alcohol dependence: A prospective study. Occupational and Environmental Medicine, 61(4), 219- 224.PubMedCrossRefGoogle Scholar
  11. Kivimäki, M., Leino-Arjas, P., Luukkonen, R., Riihimaki, H., Vahtera, J., & Kirjonen, J. (2002). Work stress and risk of cardiovascular mortality: Prospective cohort study of industrial employees. British Medical Journal, 325(7369), 857–860.PubMedCrossRefGoogle Scholar
  12. Martin, C. S., Chung, T., Kirisci, L., & Langenbucher, J. W. (2006). Item response theory analysis of diagnostic criteria for alco hol and cannabis use disorders in adolescents: Implications for DSM-V. Journal of Abnormal Psychology, 115 (4), 807–814.PubMedCrossRefGoogle Scholar
  13. Masters, G. N. (1988). The analysis of partial credit scoring. Applied Measurement in Education, 1(4), 279–297.CrossRefGoogle Scholar
  14. McDowell, I. (2006). Recommendations and conclusions. In I. McDowell (Ed.), Measuring health: A guide to rating scales and questionnaires (3rd ed., pp. 704–710). New York: Oxford University Press.Google Scholar
  15. Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16(2), 159–176.CrossRefGoogle Scholar
  16. Muraki, E., & Bock, R. D. (1993). PARSCALE: IRT-based test scoring and item analysis for graded open-ended exercises and performance tasks. Chicago: Scientific Software Int.Google Scholar
  17. Orlando, M. Critical issues to address when applying Item Response Theory (IRT) models. Retrieved Feb 24, 2007, from Scholar
  18. Peter, R., Alfredsson, L., Hammar, N., Siegrist, J., Theorell, T. E., & Westerholm, P. (1998). High effort, low reward, and cardiovascular risk factors in employed Swedish men and women: Baseline results from the WOLF Study. Journal of Epidemiology and Community Health, 52, 540–547.PubMedGoogle Scholar
  19. Pikhart, H., Bobak, M., Siegrist, J., Pajak, A., Rywik, S., Kyshegyi, J., et al. (2001). Psychosocial work characteristics and self-rated health in four post-communist countries. Journal of Epidemiology and Community Health, 55(9), 624–630.PubMedCrossRefGoogle Scholar
  20. Raczek, A., Ware, J., Bjorner, J., Gandek, B., Haley, S., Aaronson, N., et al. (1998). Comparison of Rasch and summated rating scales constructed from SF-36 physical functioning items in seven countries: Results from the IQOLA Project: International Quality of Life Assessment. Journal of Clinical Epidemiology, 51(11), 1203–1214.PubMedCrossRefGoogle Scholar
  21. Radloff, L. S. (1977). The CES-D Scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1, 385–401.CrossRefGoogle Scholar
  22. Samejima, F. (1974). Normal ogive model on the continuous response level in the multidimensional latent space. Psychome-trika,39, 111–121.CrossRefGoogle Scholar
  23. Schultz-Larsen, K., Kreiner, S., & Lomholt, R. K. (2007). Mini- mental status examination: Mixed Rasch model item analysis derived two different cognitive dimensions of the MMSE. Journal of Clinical Epidemiology, 60(3), 268–279.PubMedCrossRefGoogle Scholar
  24. Siegrist, J. (1996). Adverse health effects of high-effort/low-reward conditions. Journal of Occupational Health Psychology, 1(1), 27–41.PubMedCrossRefGoogle Scholar
  25. Siegrist, J., & Peter, R. (1997). Measuring effort-reward imbalance at work: Guidelines. Dusseldorf: Heinrich Heine University.Google Scholar
  26. Siegrist, J., Starke, D., Chandola, T., Godin, I., Marmot, M., Niedhammer, I., et al. (2004). The measurement of effort- reward imbalance at work: European comparisons. Social Science & Medicine, 58(8), 1483–1499.CrossRefGoogle Scholar
  27. Stansfeld, S. A., Fuhrer, R., Shipley, M. J., & Marmot, M. G. (1999). Work characteristics predict psychiatric disorder: Prospective results from the Whitehall II study. Occupational and Environmental Medicine, 56, 302–307.PubMedCrossRefGoogle Scholar
  28. Takane, Y., & de Leeuw, J. (1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52(3), 393–408.CrossRefGoogle Scholar
  29. Tsutsumi, A. (2004). Report of Research Project, Grand-In-Aid For Scientific Research (C), 2001-2004: Development and application of the Japanese version of effort-reward imbalance questionnaire (Project no. 12670373). Okayama: Okayama University School of Medicine and Dentistry, Hygiene, & Preventive Medicine.Google Scholar
  30. Ware, J. E., Konsinski, M., & Bjorner, J. B. (2002). SF-36 Health Survey:Manual & interpretation guide. Boston, MA: The Health Institute/New England Medical Center.Google Scholar

Copyright information

© International Society of Behavioral Medicine 2008

Authors and Affiliations

  • Akizumi Tsutsumi
    • 1
  • Noboru Iwata
    • 2
    • 1
  • Takafumi Wakita
    • 3
    • 1
  • Ryuichi Kumagai
    • 4
    • 1
  • Hiroyuki Noguchi
    • 5
    • 1
  • Norito Kawakami
    • 6
  1. 1.University of Occupational and Environmental Health, Occupational Health Training CenterKitakyushuJapan
  2. 2.Department of Clinical PsychologyHiroshima International UniversityKitakyushuJapan
  3. 3.Department of Epidemiology and Health-Care ResearchKyoto UniversityKitakyushuJapan
  4. 4.Institute of Undergraduate Programs and CoursesNiigata UniversityKitakyushuJapan
  5. 5.Department of Psychology and Human Developmental SciencesGraduate School of Education and Human Development, Nagoya UniversityKitakyushuJapan
  6. 6.Department of Mental HealthUniversity of Tokyo Graduate School of MedicineJapan

Personalised recommendations