Description and application of a method to quantify criterion-related cut-off values for questionnaire-based psychosocial risk assessment

Abstract

Purpose

The psychosocial risk assessment is a systematic intervention process for organizations that aims at improving psychosocial working conditions as well as employee health. Based on a screening of working conditions, interventions to reduce risk factors are implemented and evaluated. What is missing for most screening instruments however are cut-off values to categorize working conditions into uncritical vs. critical, whereas the latter indicates an elevated risk for illness. To estimate and evaluate cut-off values, two studies were conducted using the receiver operating characteristic (ROC) analysis.

Methods

In Study 1, a sample of 229 participants answered an online survey on depression (PHQ) and psychosocial working conditions using a questionnaire (DYNAMIK) that covers five factors important to workers' health: workload, boundary permeability, participation, leader support, and usability. Using the ROC analysis, criterion-related cut-off values were generated to predict depressive symptoms. In Study 2, these cut-off values were used to classify working conditions in the two categories of ‘critical’ and ‘uncritical’ in an independent sample (N = 295). It was tested for differences in the results of the two groups concerning the direct criterion of depressive symptoms and the indirect criterion of effort-reward imbalance.

Results

In Study 1, cut-off values differed between the five scales and showed different values for sensitivity and specificity. In Study 2, participants exposed to critical working conditions reported more depressive symptoms as well as an effort-reward imbalance.

Conclusions

Cut-off values are useful to identify working conditions as either critical or uncritical. This knowledge is important when deciding which working conditions should be optimized within the context of psychosocial risk assessment.

This is a preview of subscription content, access via your institution.

Fig. 1

References

  1. Altman DG, Bland JM (1994) Diagnostic tests 3: receiver operating characteristic plots. Br Med J 309:188

    CAS  Article  Google Scholar 

  2. Amstad FT, Meier LL, Fasel U, Elfering A, Semmer NK (2011) A meta-analysis of work-family conflict and various outcomes with a special emphasis on cross-domain versus matching-domain relations. J Occup Health Psychol 16:151–169. https://doi.org/10.1037/a0022170

    Article  Google Scholar 

  3. Barth J, Schumacher M, Herrmann-Lingen C (2004) Depression as a risk factor for mortality in patients with coronary heart disease: a meta-analysis. Psychosom Med 66:802–813. https://doi.org/10.1097/01.psy.0000146332.53619.b2

    Article  Google Scholar 

  4. Beck D, Lenhardt U (2019) Consideration of psychosocial factors in workplace risk assessments: findings from a company survey in Germany. Int Arch Occup Environ Health 92:435–451. https://doi.org/10.1007/s00420-019-01416-5

    Article  Google Scholar 

  5. Behar E, Alcaine O, Zuellig AR, Borkovec TD (2003) Screening for generalized anxiety disorder using the Penn State Worry Questionnaire: a receiver operating characteristic analysis. J Behav Ther Exp Psychol 34:25–43. https://doi.org/10.1016/S0005-7916(03)00004-1

    Article  Google Scholar 

  6. Bonde JPE (2008) Psychosocial factors at work and risk of depression: a systematic review of the epidemiological evidence. Occup Environ Med 65:438–445. https://doi.org/10.1136/oem.2007.038430

    CAS  Article  Google Scholar 

  7. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Erlbaum, Hillsdale

    Google Scholar 

  8. Diebig M, Jungmann F, Müller A, Wulf IC (2018) Inhalts- und prozessbezogene Anforderungen an die Gefährdungsbeurteilung psychischer Belastung im Kontext Industrie 4.0. Z Arb Organ 62:53–67. https://doi.org/10.1026/0932-4089/a000265

    Article  Google Scholar 

  9. Diebig M, Dragano N, Körner U, Lunau T, Wulf IC, Angerer P (2020) Development and validation of a questionnaire to measure psychosocial work stressors in modern working environments. J Occup Environ Med 62:185–193. https://doi.org/10.1097/JOM.0000000000001779

    Article  Google Scholar 

  10. Dragano N, Siegrist J, Nyberg ST, Lunau T, Fransson EI, Alfredsson L, Bjorner JB, Borritz M, Burr H, Erbel R, Fahlén G, Goldberg M, Hamer M, Heikkilä K, Jöckel K-H, Knutsson A, Madsen IEH, Nielsen ML, Nordin M, Oksanen T, Pejtersen JH, Pentti J, Rugulies R, Salo P, Schupp J, Singh-Manoux A, Steptoe A, Theorell T, Vahtera J, Westerholm PJM, Westerlund H, Virtanen M, Zins M, Batty GD, Kivimäki M (2017) Effort-reward imbalance at work and incident coronary heart disease: a multicohort study of 90,164 individuals. Epidemiology 28:619–626. https://doi.org/10.1097/EDE.0000000000000666

    Article  Google Scholar 

  11. Eddy P, Heckenberg R, Wertheim EH, Kent S, Wright BJ (2016) A systematic review and meta-analysis of the effort-reward imbalance model of workplace stress with indicators of immune function. J Psychosom Res 91:1–8. https://doi.org/10.1016/j.jpsychores.2016.10.003

    Article  Google Scholar 

  12. Eddy P, Wertheim EH, Kingsley M, Wright BJ (2017) Associations between the effort-reward imbalance model of workplace stress and indices of cardiovascular health: a systematic review and meta-analysis. Neurosci Biobehav Res 83:252–266. https://doi.org/10.1016/j.neubiorev.2017.10.025

    Article  Google Scholar 

  13. Eng J (2005) Receiver operating characteristic analysis: a primer. Acad Radiol 12:909–916. https://doi.org/10.1016/j.acra.2005.04.005

    Article  Google Scholar 

  14. European Agency for Safety and Health at Work (2014) E-guide to managing psychosocial risks. European Communities, Luxembourg

  15. Follmer KB, Jones KS (2018) Mental illness in the workplace: an interdisciplinary review and organizational research agenda. J Manag 44:325–351. https://doi.org/10.1177/0149206317741194

    Article  Google Scholar 

  16. Ganster DC, Rosen CC (2013) Work stress and employee health: a multidisciplinary review. J Manag 39:1085–1122. https://doi.org/10.1177/0149206313475815

    Article  Google Scholar 

  17. Hapke U, Cohrdes C, Nübel J (2019) Depressive symptoms in a European comparison—results from the European Health Interview Survey (EHIS) 2. J Health Monit 4:57–64. https://doi.org/10.25646/6223

    Article  Google Scholar 

  18. Juvani A, La Oksanen T, Virtanen M, Salo P, Pentti J, Kivimäki M, Vahtera J (2018) Clustering of job strain, effort-reward imbalance, and organizational injustice and the risk of work disability: a cohort study. Scand J Work Environ Health 44:485–495. https://doi.org/10.5271/sjweh.3736

    Article  Google Scholar 

  19. Karasek RA (1979) Job demands, job decision latitude, and mental strain: Implications for job redesign. Admin Sci Q 24:285–308. https://doi.org/10.2307/2392498

    Article  Google Scholar 

  20. Kivimäki M, Virtanen M, Elovainio M, Kouvonen A, Väänänen A, Vahtera J (2006) Work stress in the etiology of coronary heart disease—a meta-analysis. Scand J Work Environ Health 32:431–442

    Article  Google Scholar 

  21. Kivimäki M, Pentti J, Ferrie JE, Batty GD, Nyberg ST, Jokela M, Virtanen M, Alfredsson L, Dragano N, Fransson EI, Goldberg M, Knutsson A, Koskenvuo M, Koskinen A, Kouvonen A, Luukkonen R, Oksanen T, Rugulies R, Siegrist J, Singh-Manoux A, Suominen S, Theorell T, Väänänen A, Vahtera J, Westerholm PJM, Westerlund H, Zins M, Strandberg T, Steptoe A, Deanfield J (2018) Work stress and risk of death in men and women with and without cardiometabolic disease: a multicohort study. Lancet Diabetes Endocrinol 6:705–713. https://doi.org/10.1016/S2213-8587(18)30140-2

    Article  Google Scholar 

  22. Kocalevent R-D, Hinz A, Brähler E (2013) Standardization of the depression screener patient health questionnaire (PHQ-9) in the general population. Gen Hosp Psychiatry 35:551–555. https://doi.org/10.1016/j.genhosppsych.2013.04.006

    Article  Google Scholar 

  23. Kroenke K, Spitzer RL, Williams JB (2001) The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med 16:606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x

    CAS  Article  Google Scholar 

  24. Kroenke K, Spitzer RL, Williams JB (2003) The Patient Health Questionnaire-2: validity of a two-item depression screener. Med Care 41:1284–1292. https://doi.org/10.1097/01.MLR.0000093487.78664.3C

    Article  Google Scholar 

  25. Kroenke K, Spitzer RL, Williams JB, Löwe B (2010) The patient health questionnaire somatic, anxiety, and depressive symptom scales: a systematic review. Gen Hosp Psychiatry 32:345–359. https://doi.org/10.1016/j.genhosppsych.2010.03.006

    Article  Google Scholar 

  26. Lehr D, Koch S, Hillert A (2010) Where is (im)balance? Necessity and construction of evaluated cut-off points for effort-reward imbalance and overcommitment. J Occup Organ Psychol 83:251–261. https://doi.org/10.1348/096317909X406772

    Article  Google Scholar 

  27. Leka S, Jain A, Iavicoli S, Di Tecco C (2015) An evaluation of the policy context on psychosocial risks and mental health in the workplace in the European Union: achievements, challenges, and the future. Biomed Res Int 2015:1–18. https://doi.org/10.1155/2015/213089

    CAS  Article  Google Scholar 

  28. Lopez AD, Mathers CD, Ezzati M, Jamison DT, Murray CJL (2006) Global and regional burden of disease and risk factors, 2001: systematic analysis of population health data. Lancet 367:1747–1757. https://doi.org/10.1016/S0140-6736(06)68770-9

    Article  Google Scholar 

  29. Löwe B, Kroenke K, Herzog W, Gräfe K (2004) Measuring depression outcome with a brief self-report instrument: sensitivity to change of the Patient Health Questionnaire (PHQ-9). J Affect Disord 81:61–66. https://doi.org/10.1016/S0165-0327(03)00198-8

    Article  Google Scholar 

  30. Manea L, Gilbody S, Hewitt C, North A, Plummer F, Richardson R, Thombs BD, Williams B, McMillan D (2016) Identifying depression with the PHQ-2: a diagnostic meta-analysis. J Affect Disord 203:382–395. https://doi.org/10.1016/j.jad.2016.06.003

    Article  Google Scholar 

  31. Nielsen K, Randall R, Holten A-L, González ER (2010) Conducting organizational-level occupational health interventions: what works? Work Stress 24:234–259. https://doi.org/10.1080/02678373.2010.515393

    Article  Google Scholar 

  32. Podsakoff PM, MacKenzie SB, Lee J-Y, Podsakoff NP (2003) Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol 88:879–903. https://doi.org/10.1037/0021-9010.88.5.879

    Article  Google Scholar 

  33. R Core Team (2018) R: a language and environment for statistical computing. R foundation for statistical computing, Vienna, Austria. https://www.R-project.org/

  34. Richiardi L, Pizzi C, Pearce N (2013) Commentary: representativeness is usually not necessary and often should be avoided. Int J Epidemiol 42:1018–1022. https://doi.org/10.1093/ije/dyt103

    Article  Google Scholar 

  35. Rick J, Briner RB (2000) Psychosocial risk assessment: problems and prospects. Occup Med-C 50:310–314. https://doi.org/10.1093/occmed/50.5.310

    CAS  Article  Google Scholar 

  36. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez J-C, Müller M (2011) pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinform 12:77. https://doi.org/10.1186/1471-2105-12-77

    Article  Google Scholar 

  37. Rothman KJ, Gallacher JEJ, Hatch EE (2013) Why representativeness should be avoided. Int J Epidemiol 42:1012–1014. https://doi.org/10.1093/ije/dys223

    Article  Google Scholar 

  38. Rugulies R, Aust B, Madsen IEH (2017) Effort–reward imbalance at work and risk of depressive disorders. A systematic review and meta-analysis of prospective cohort studies. Scand J Work Environ Health 43:294–306

    Article  Google Scholar 

  39. Schmidt S, Roesler U, Kusserow T, Rau R (2014) Uncertainty in the workplace: examining role ambiguity and role conflict, and their link to depression—a meta-analysis. Eur J Work Organ Psychol 23:91–106. https://doi.org/10.1080/1359432X.2012.711523

    Article  Google Scholar 

  40. Siegrist J (1996) Adverse health effects of high-effort/low-reward conditions. J Occup Health Psychol 1:27–41. https://doi.org/10.1037/1076-8998.1.1.27

    CAS  Article  Google Scholar 

  41. Siegrist J, Starke D, Chandola T, Godin I, Marmot M, Niedhammer I, Peter R (2004) The measurement of effort–reward imbalance at work: European comparisons. Soc Sci Med 58:1483–1499. https://doi.org/10.1016/S0277-9536(03)00351-4

    Article  Google Scholar 

  42. Siegrist J, Wege N, Pühlhofer F, Wahrendorf M (2009) A short generic measure of work stress in the era of globalization: effort-reward imbalance. Int Arch Occup Environ Health 82:1005–1013. https://doi.org/10.1007/s00420-008-0384-3

    Article  Google Scholar 

  43. Stansfeld S, Candy B (2006) Psychosocial work environment and mental health—a meta-analytic review. Scand J Work Environ Health 32:443–462

    Article  Google Scholar 

  44. Swets JA (1988) Measuring the accuracy of diagnostic systems. Science 240:1285–1293. https://doi.org/10.1126/science.3287615

    CAS  Article  Google Scholar 

  45. Zweig MH, Campbell G (1993) Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem 39:561–577. https://doi.org/10.1093/clinchem/39.4.561

    CAS  Article  Google Scholar 

Download references

Funding

This work was funded by the German Federal Ministry of Education and Research (BMBF) as part of the program entitled ‘Innovations for Tomorrow's Production, Services, and Work’ under Grant FKZ 02L14A170.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Mathias Diebig.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

The project has been approved by the ethics committee at the Medical Faculty of Heinrich Heine University Düsseldorf (No. 5562) and has been performed in accordance with the ethical standards as laid down in the 2013 Declaration of Helsinki.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Content of the DYNAMIK Questionnaire
Work load
1 WL01: time pressure (+)
2 WL02: interruption of work (+)
3 WL03: multi-tasking (+)
4 WL04: flexibility requirements (+)
Boundary permeability
1 BP01: extensive overtime (+)
2 BP02: insufficient breaks (−)
3 BP03: work-family balance (+)
4 BP04: work during leisure time (+)
Participation
1 PN01: participation in decision-making (+)
2 PN02: influence on work content (+)
3 PN03: influence on work methods/procedures (+)
Leader support
1 LS01: conflict with the leader (−)
2 LS02: support by the leader (+)
3 LS03: recognition of work performance (+)
Usability
1 USB01: technical problems (−)
2 USB02: usability (+)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Diebig, M., Angerer, P. Description and application of a method to quantify criterion-related cut-off values for questionnaire-based psychosocial risk assessment. Int Arch Occup Environ Health 94, 475–485 (2021). https://doi.org/10.1007/s00420-020-01597-4

Download citation

Keywords

  • Psychosocial risk assessment
  • Psychosocial factors
  • Depression
  • Workplace risk assessment
  • Occupational safety and health