The psychosocial risk assessment is a systematic intervention process for organizations that aims at improving psychosocial working conditions as well as employee health. Based on a screening of working conditions, interventions to reduce risk factors are implemented and evaluated. What is missing for most screening instruments however are cut-off values to categorize working conditions into uncritical vs. critical, whereas the latter indicates an elevated risk for illness. To estimate and evaluate cut-off values, two studies were conducted using the receiver operating characteristic (ROC) analysis.
In Study 1, a sample of 229 participants answered an online survey on depression (PHQ) and psychosocial working conditions using a questionnaire (DYNAMIK) that covers five factors important to workers' health: workload, boundary permeability, participation, leader support, and usability. Using the ROC analysis, criterion-related cut-off values were generated to predict depressive symptoms. In Study 2, these cut-off values were used to classify working conditions in the two categories of ‘critical’ and ‘uncritical’ in an independent sample (N = 295). It was tested for differences in the results of the two groups concerning the direct criterion of depressive symptoms and the indirect criterion of effort-reward imbalance.
In Study 1, cut-off values differed between the five scales and showed different values for sensitivity and specificity. In Study 2, participants exposed to critical working conditions reported more depressive symptoms as well as an effort-reward imbalance.
Cut-off values are useful to identify working conditions as either critical or uncritical. This knowledge is important when deciding which working conditions should be optimized within the context of psychosocial risk assessment.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Altman DG, Bland JM (1994) Diagnostic tests 3: receiver operating characteristic plots. Br Med J 309:188
Amstad FT, Meier LL, Fasel U, Elfering A, Semmer NK (2011) A meta-analysis of work-family conflict and various outcomes with a special emphasis on cross-domain versus matching-domain relations. J Occup Health Psychol 16:151–169. https://doi.org/10.1037/a0022170
Barth J, Schumacher M, Herrmann-Lingen C (2004) Depression as a risk factor for mortality in patients with coronary heart disease: a meta-analysis. Psychosom Med 66:802–813. https://doi.org/10.1097/01.psy.0000146332.53619.b2
Beck D, Lenhardt U (2019) Consideration of psychosocial factors in workplace risk assessments: findings from a company survey in Germany. Int Arch Occup Environ Health 92:435–451. https://doi.org/10.1007/s00420-019-01416-5
Behar E, Alcaine O, Zuellig AR, Borkovec TD (2003) Screening for generalized anxiety disorder using the Penn State Worry Questionnaire: a receiver operating characteristic analysis. J Behav Ther Exp Psychol 34:25–43. https://doi.org/10.1016/S0005-7916(03)00004-1
Bonde JPE (2008) Psychosocial factors at work and risk of depression: a systematic review of the epidemiological evidence. Occup Environ Med 65:438–445. https://doi.org/10.1136/oem.2007.038430
Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Erlbaum, Hillsdale
Diebig M, Jungmann F, Müller A, Wulf IC (2018) Inhalts- und prozessbezogene Anforderungen an die Gefährdungsbeurteilung psychischer Belastung im Kontext Industrie 4.0. Z Arb Organ 62:53–67. https://doi.org/10.1026/0932-4089/a000265
Diebig M, Dragano N, Körner U, Lunau T, Wulf IC, Angerer P (2020) Development and validation of a questionnaire to measure psychosocial work stressors in modern working environments. J Occup Environ Med 62:185–193. https://doi.org/10.1097/JOM.0000000000001779
Dragano N, Siegrist J, Nyberg ST, Lunau T, Fransson EI, Alfredsson L, Bjorner JB, Borritz M, Burr H, Erbel R, Fahlén G, Goldberg M, Hamer M, Heikkilä K, Jöckel K-H, Knutsson A, Madsen IEH, Nielsen ML, Nordin M, Oksanen T, Pejtersen JH, Pentti J, Rugulies R, Salo P, Schupp J, Singh-Manoux A, Steptoe A, Theorell T, Vahtera J, Westerholm PJM, Westerlund H, Virtanen M, Zins M, Batty GD, Kivimäki M (2017) Effort-reward imbalance at work and incident coronary heart disease: a multicohort study of 90,164 individuals. Epidemiology 28:619–626. https://doi.org/10.1097/EDE.0000000000000666
Eddy P, Heckenberg R, Wertheim EH, Kent S, Wright BJ (2016) A systematic review and meta-analysis of the effort-reward imbalance model of workplace stress with indicators of immune function. J Psychosom Res 91:1–8. https://doi.org/10.1016/j.jpsychores.2016.10.003
Eddy P, Wertheim EH, Kingsley M, Wright BJ (2017) Associations between the effort-reward imbalance model of workplace stress and indices of cardiovascular health: a systematic review and meta-analysis. Neurosci Biobehav Res 83:252–266. https://doi.org/10.1016/j.neubiorev.2017.10.025
Eng J (2005) Receiver operating characteristic analysis: a primer. Acad Radiol 12:909–916. https://doi.org/10.1016/j.acra.2005.04.005
European Agency for Safety and Health at Work (2014) E-guide to managing psychosocial risks. European Communities, Luxembourg
Follmer KB, Jones KS (2018) Mental illness in the workplace: an interdisciplinary review and organizational research agenda. J Manag 44:325–351. https://doi.org/10.1177/0149206317741194
Ganster DC, Rosen CC (2013) Work stress and employee health: a multidisciplinary review. J Manag 39:1085–1122. https://doi.org/10.1177/0149206313475815
Hapke U, Cohrdes C, Nübel J (2019) Depressive symptoms in a European comparison—results from the European Health Interview Survey (EHIS) 2. J Health Monit 4:57–64. https://doi.org/10.25646/6223
Juvani A, La Oksanen T, Virtanen M, Salo P, Pentti J, Kivimäki M, Vahtera J (2018) Clustering of job strain, effort-reward imbalance, and organizational injustice and the risk of work disability: a cohort study. Scand J Work Environ Health 44:485–495. https://doi.org/10.5271/sjweh.3736
Karasek RA (1979) Job demands, job decision latitude, and mental strain: Implications for job redesign. Admin Sci Q 24:285–308. https://doi.org/10.2307/2392498
Kivimäki M, Virtanen M, Elovainio M, Kouvonen A, Väänänen A, Vahtera J (2006) Work stress in the etiology of coronary heart disease—a meta-analysis. Scand J Work Environ Health 32:431–442
Kivimäki M, Pentti J, Ferrie JE, Batty GD, Nyberg ST, Jokela M, Virtanen M, Alfredsson L, Dragano N, Fransson EI, Goldberg M, Knutsson A, Koskenvuo M, Koskinen A, Kouvonen A, Luukkonen R, Oksanen T, Rugulies R, Siegrist J, Singh-Manoux A, Suominen S, Theorell T, Väänänen A, Vahtera J, Westerholm PJM, Westerlund H, Zins M, Strandberg T, Steptoe A, Deanfield J (2018) Work stress and risk of death in men and women with and without cardiometabolic disease: a multicohort study. Lancet Diabetes Endocrinol 6:705–713. https://doi.org/10.1016/S2213-8587(18)30140-2
Kocalevent R-D, Hinz A, Brähler E (2013) Standardization of the depression screener patient health questionnaire (PHQ-9) in the general population. Gen Hosp Psychiatry 35:551–555. https://doi.org/10.1016/j.genhosppsych.2013.04.006
Kroenke K, Spitzer RL, Williams JB (2001) The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med 16:606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x
Kroenke K, Spitzer RL, Williams JB (2003) The Patient Health Questionnaire-2: validity of a two-item depression screener. Med Care 41:1284–1292. https://doi.org/10.1097/01.MLR.0000093487.78664.3C
Kroenke K, Spitzer RL, Williams JB, Löwe B (2010) The patient health questionnaire somatic, anxiety, and depressive symptom scales: a systematic review. Gen Hosp Psychiatry 32:345–359. https://doi.org/10.1016/j.genhosppsych.2010.03.006
Lehr D, Koch S, Hillert A (2010) Where is (im)balance? Necessity and construction of evaluated cut-off points for effort-reward imbalance and overcommitment. J Occup Organ Psychol 83:251–261. https://doi.org/10.1348/096317909X406772
Leka S, Jain A, Iavicoli S, Di Tecco C (2015) An evaluation of the policy context on psychosocial risks and mental health in the workplace in the European Union: achievements, challenges, and the future. Biomed Res Int 2015:1–18. https://doi.org/10.1155/2015/213089
Lopez AD, Mathers CD, Ezzati M, Jamison DT, Murray CJL (2006) Global and regional burden of disease and risk factors, 2001: systematic analysis of population health data. Lancet 367:1747–1757. https://doi.org/10.1016/S0140-6736(06)68770-9
Löwe B, Kroenke K, Herzog W, Gräfe K (2004) Measuring depression outcome with a brief self-report instrument: sensitivity to change of the Patient Health Questionnaire (PHQ-9). J Affect Disord 81:61–66. https://doi.org/10.1016/S0165-0327(03)00198-8
Manea L, Gilbody S, Hewitt C, North A, Plummer F, Richardson R, Thombs BD, Williams B, McMillan D (2016) Identifying depression with the PHQ-2: a diagnostic meta-analysis. J Affect Disord 203:382–395. https://doi.org/10.1016/j.jad.2016.06.003
Nielsen K, Randall R, Holten A-L, González ER (2010) Conducting organizational-level occupational health interventions: what works? Work Stress 24:234–259. https://doi.org/10.1080/02678373.2010.515393
Podsakoff PM, MacKenzie SB, Lee J-Y, Podsakoff NP (2003) Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol 88:879–903. https://doi.org/10.1037/0021-9010.88.5.879
R Core Team (2018) R: a language and environment for statistical computing. R foundation for statistical computing, Vienna, Austria. https://www.R-project.org/
Richiardi L, Pizzi C, Pearce N (2013) Commentary: representativeness is usually not necessary and often should be avoided. Int J Epidemiol 42:1018–1022. https://doi.org/10.1093/ije/dyt103
Rick J, Briner RB (2000) Psychosocial risk assessment: problems and prospects. Occup Med-C 50:310–314. https://doi.org/10.1093/occmed/50.5.310
Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez J-C, Müller M (2011) pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinform 12:77. https://doi.org/10.1186/1471-2105-12-77
Rothman KJ, Gallacher JEJ, Hatch EE (2013) Why representativeness should be avoided. Int J Epidemiol 42:1012–1014. https://doi.org/10.1093/ije/dys223
Rugulies R, Aust B, Madsen IEH (2017) Effort–reward imbalance at work and risk of depressive disorders. A systematic review and meta-analysis of prospective cohort studies. Scand J Work Environ Health 43:294–306
Schmidt S, Roesler U, Kusserow T, Rau R (2014) Uncertainty in the workplace: examining role ambiguity and role conflict, and their link to depression—a meta-analysis. Eur J Work Organ Psychol 23:91–106. https://doi.org/10.1080/1359432X.2012.711523
Siegrist J (1996) Adverse health effects of high-effort/low-reward conditions. J Occup Health Psychol 1:27–41. https://doi.org/10.1037/1076-8918.104.22.168
Siegrist J, Starke D, Chandola T, Godin I, Marmot M, Niedhammer I, Peter R (2004) The measurement of effort–reward imbalance at work: European comparisons. Soc Sci Med 58:1483–1499. https://doi.org/10.1016/S0277-9536(03)00351-4
Siegrist J, Wege N, Pühlhofer F, Wahrendorf M (2009) A short generic measure of work stress in the era of globalization: effort-reward imbalance. Int Arch Occup Environ Health 82:1005–1013. https://doi.org/10.1007/s00420-008-0384-3
Stansfeld S, Candy B (2006) Psychosocial work environment and mental health—a meta-analytic review. Scand J Work Environ Health 32:443–462
Swets JA (1988) Measuring the accuracy of diagnostic systems. Science 240:1285–1293. https://doi.org/10.1126/science.3287615
Zweig MH, Campbell G (1993) Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem 39:561–577. https://doi.org/10.1093/clinchem/39.4.561
This work was funded by the German Federal Ministry of Education and Research (BMBF) as part of the program entitled ‘Innovations for Tomorrow's Production, Services, and Work’ under Grant FKZ 02L14A170.
Conflict of interest
The authors declare that they have no conflict of interest.
The project has been approved by the ethics committee at the Medical Faculty of Heinrich Heine University Düsseldorf (No. 5562) and has been performed in accordance with the ethical standards as laid down in the 2013 Declaration of Helsinki.
Informed consent was obtained from all individual participants included in the study.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
|Content of the DYNAMIK Questionnaire|
|1||WL01: time pressure (+)|
|2||WL02: interruption of work (+)|
|3||WL03: multi-tasking (+)|
|4||WL04: flexibility requirements (+)|
|1||BP01: extensive overtime (+)|
|2||BP02: insufficient breaks (−)|
|3||BP03: work-family balance (+)|
|4||BP04: work during leisure time (+)|
|1||PN01: participation in decision-making (+)|
|2||PN02: influence on work content (+)|
|3||PN03: influence on work methods/procedures (+)|
|1||LS01: conflict with the leader (−)|
|2||LS02: support by the leader (+)|
|3||LS03: recognition of work performance (+)|
|1||USB01: technical problems (−)|
|2||USB02: usability (+)|
About this article
Cite this article
Diebig, M., Angerer, P. Description and application of a method to quantify criterion-related cut-off values for questionnaire-based psychosocial risk assessment. Int Arch Occup Environ Health 94, 475–485 (2021). https://doi.org/10.1007/s00420-020-01597-4
- Psychosocial risk assessment
- Psychosocial factors
- Workplace risk assessment
- Occupational safety and health