Measures of Workplace Inclusion: A Systematic Review Using the COSMIN Methodology

  • Mana RezaiEmail author
  • Kendall Kolne
  • Sunny Bui
  • Sally Lindsay


Purpose To systematically assess the measurement properties and the quality of the evidence for measures of inclusion or exclusion at work. Methods Comprehensive searches of five electronic databases were conducted up to February 2019. Eligible studies aimed to develop a measure of workplace inclusion or exclusion or assessed at least one measurement property. Pairs of reviewers independently screened articles and assessed risk of bias. Methodological quality was appraised with the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist. A best-evidence synthesis approach guided the analysis. For each measurement property, evidence quality was rated as high, moderate, low, or very low and results were classified as sufficient, insufficient, or inconsistent. Results The titles and abstracts of 14,380 articles were screened, with 151 full-text articles reviewed for eligibility. Of these, 27 studies were identified, 10 of which were measure development studies. Included measures were the Workplace Ostracism Scale, Ostracism Interventionary Behaviour Scale, Workplace Culture Survey, Workplace Exclusion Scale, Perceived Group Inclusion Scale, Organizational Cultural Intelligence Scale, Inclusion–Exclusion Scale, Climate for Inclusion Scale, Workplace Social Inclusion Scale and the Inclusion-Diversity Scale. Most workplace inclusion instruments were not examined for some form of validity or reliability and evidence for responsiveness was absent. The quality of the evidence for content validity was low for 30% of studies and very low for 70% of studies. Conclusion Future research should focus on comprehensive evaluations of the psychometric properties of existing measures, with an emphasis on content validity, measurement error, reliability and responsiveness.


Work Discrimination Social participation Psychometrics Systematic review 



This study was funded in part by the Canadian Institutes of Health Research-Social Sciences and Humanities Research Council (CIHR-SSHRC) Partnership Grant (01561-000 and 895-2018-4002) awarded to Sally Lindsay and the Kimel Family Fund through the Holland Bloorview Kids Rehabilitation Hospital.

Author Contributions

Mana Rezai and Sally Lindsay contributed to the study conception and design. All authors contributed to the literature search and data extraction. Analyses were primarily performed by Mana Rezai, and Kendall Kolne. The first draft of the manuscript was written by Mana Rezai and all authors commented on subsequent versions of the manuscript. All authors read and approved the final manuscript.

Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

No animal or human studies were carried out by the authors for this article.

Supplementary material

10926_2020_9872_MOESM1_ESM.pdf (260 kb)
Supplementary file1 (PDF 259 kb)
10926_2020_9872_MOESM2_ESM.pdf (286 kb)
Supplementary file2 (PDF 286 kb)
10926_2020_9872_MOESM3_ESM.pdf (166 kb)
Supplementary file3 (PDF 166 kb)
10926_2020_9872_MOESM4_ESM.pdf (51 kb)
Supplementary file4 (PDF 51 kb)
10926_2020_9872_MOESM5_ESM.pdf (268 kb)
Supplementary file5 (PDF 267 kb)
10926_2020_9872_MOESM6_ESM.pdf (436 kb)
Supplementary file6 (PDF 435 kb)
10926_2020_9872_MOESM7_ESM.pdf (300 kb)
Supplementary file7 (PDF 300 kb)


  1. 1.
    Bethge M, Radoschewski FM. Physical and psychosocial work stressors, health-related control beliefs and work ability: cross-sectional findings from the German Sociomedical Panel of Employees. Int Arch Occup Environ Health. 2010;83(3):241–250.PubMedCrossRefGoogle Scholar
  2. 2.
    Nishii LH. The benefits of climate for inclusion for gender-diverse groups. Acad Manage J. 2013;56(6):1754–1774.CrossRefGoogle Scholar
  3. 3.
    Mor Barak ME, Cherin DA. A tool to expand organizational understanding of workforce diversity: exploring a measure of inclusion-exclusion. Adm Soc Work. 1998;22(1):47–644.CrossRefGoogle Scholar
  4. 4.
    Benach J, Vives A, Amable M, Vanroelen C, Tarafa G, Muntaner C. Precarious employment: understanding an emerging social determinant of health. Annu Rev Public Health. 2014;35:229–253.PubMedCrossRefGoogle Scholar
  5. 5.
    Benach J, Muntaner C, Santana V. EMCONET (employment conditions knowledge network). Employment conditions and health inequalities: final report to the WHO commission on social determinants of health (CSDH); 2007. p. 172.Google Scholar
  6. 6.
    Greenglass E, Fiksenbaum L, Burke RJ. Components of social support, buffering effects and burnout: implications for psychological functioning. Anxiety Stress Coping. 1996;9(3):185–197.CrossRefGoogle Scholar
  7. 7.
    Mor Barak ME, Cherin DA, Berkman S. Organizational and personal dimensions in diversity climate: ethnic and gender differences in employee perceptions. J Appl Behav Sci. 1998;34(1):82–104.CrossRefGoogle Scholar
  8. 8.
    Schaufeli WB, Dierendonck DV, Gorp KV. Burnout and reciprocity: towards a dual-level social exchange model. Work Stress. 1996;10(3):225–237.CrossRefGoogle Scholar
  9. 9.
    Mor Barak ME, Levin A. Outside of the corporate mainstream and excluded from the work community: a study of diversity, job satisfaction and well-being. Community, Work Fam. 2002;5(2):133–157.CrossRefGoogle Scholar
  10. 10.
    Mor Barak ME. Managing diversity: toward a globally inclusive workplace. Thousand Oaks: Sage Publications; 2011.Google Scholar
  11. 11.
    Davidson MN, Ferdman BM. Inclusion: what can I and my organization do about it. Ind-Organ Psychol. 2002;39(4):80–85.Google Scholar
  12. 12.
    Tajfel H. Social psychology of intergroup relations. Annu Rev Psychol. 1982;33(1):1–39.CrossRefGoogle Scholar
  13. 13.
    Findler L, Wind LH, Mor Barak ME. The challenge of workforce management in a global society: modeling the relationship between diversity, inclusion, organizational culture, and employee well-being, job satisfaction and organizational commitment. Adm Soc Work. 2007;31(3):63–94.CrossRefGoogle Scholar
  14. 14.
    Le H, Jiang Z, Fujimoto Y, Nielsen I. Inclusion and affective well-being: roles of justice perceptions. Pers Rev. 2018;47(4):805–820.CrossRefGoogle Scholar
  15. 15.
    De Jonge J, Schaufeli WB. Job characteristics and employee well-being: a test of Warr's Vitamin Model in health care workers using structural equation modelling. J Organ Behav Int J Ind Occup Organ Psychol Behav. 1998;19(4):387–407.Google Scholar
  16. 16.
    Snow DL, Kline ML. Preventive interventions in the workplace to reduce negative psychiatric consequences of work and family stress. In: Mazure CM, editor. Does stress cause psychiatric illness?. Washington, DC: American Psychiatric Press; 1995. p. 221–270.Google Scholar
  17. 17.
    Harrison D, Sellers A. Occupation for mental health and social inclusion. Br J Occup Ther. 2008;71(5):216–219.CrossRefGoogle Scholar
  18. 18.
    Bertram G, Stickley T. Mental health nurses, promoters of inclusion or perpetuators of exclusion? J Psychiatr Ment Health Nurs. 2005;12(4):387–395.PubMedCrossRefGoogle Scholar
  19. 19.
    Leary MR, Downs DL. Interpersonal functions of the self-esteem motive. In: Kernis MH, editor. Efficacy, agency, and self-esteem. New York: Springer; 1995. p. 123–144.Google Scholar
  20. 20.
    Roberson QM. Disentangling the meanings of diversity and inclusion in organizations. Group Organ Manage. 2006;31(2):212–236.CrossRefGoogle Scholar
  21. 21.
    Shore LM, Randel AE, Chung BG, Dean MA, Holcombe Erhart K, Singh G. Inclusion and diversity in work groups: a review and model for future research. J Manage. 2011;37(4):1262–1289.Google Scholar
  22. 22.
    Brewer MB. The social self: on being the same and different at the same time. Pers Soc Psychol Bull. 1991;17(5):475–482.CrossRefGoogle Scholar
  23. 23.
    Festinger L. A theory of social comparison processes. Human Relat. 1954;7(2):117–140.CrossRefGoogle Scholar
  24. 24.
    Blau P. Power and exchange in social life. New York: Wiley; 1964.Google Scholar
  25. 25.
    Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191.PubMedCrossRefGoogle Scholar
  26. 26.
    Mor Barak ME, Findler L, Wind LH. Diversity, inclusion, and commitment in organizations: international empirical explorations. Institute of applied and behavioral management. J Behav Appl Manage. 2001;2(2):70–91.Google Scholar
  27. 27.
    United Nations Convention on the Rights of Persons with Disabilities. Accessed 7 May 2019.
  28. 28.
    Zhu X, Law KS, Sun C, Yang D. Thriving of employees with disabilities: the roles of job self-efficacy, inclusion, and team-learning climate. Hum Resour Manage. 2019;58(1):21–34.CrossRefGoogle Scholar
  29. 29.
    Lindsay S, Cagliostro E, Albarico M, Mortaji N, Karon L. A systematic review of the benefits of hiring people with disabilities. J Occup Rehabil. 2018;28(4):634–655.PubMedCrossRefGoogle Scholar
  30. 30.
    Burke J, Bezyak J, Fraser RT, Pete J, Ditchman N, Chan F. Employers' attitudes towards hiring and retaining people with disabilities: a review of the literature. Aust J Rehabil Couns. 2013;19(1):21–38.CrossRefGoogle Scholar
  31. 31.
    Green JH, Brooke V. Recruiting and retaining the best from America's largest untapped talent pool. J Vocat Rehabil. 2001;16(2):83–88.Google Scholar
  32. 32.
    Baker PM, Linden MA, LaForce SS, Rutledge J, Goughnour KP. Barriers to employment participation of individuals with disabilities: addressing the impact of employer (mis) perception and policy. Am Behav Sci. 2018;62(5):657–675.CrossRefGoogle Scholar
  33. 33.
    Coombs T, Nicholas A, Pirkis J. A review of social inclusion measures. Aust N Z J Psychiatry. 2013;47(10):906–919.PubMedCrossRefPubMedCentralGoogle Scholar
  34. 34.
    Cordier R, Milbourn B, Martin R, Buchanan A, Chung D, Speyer R. A systematic review evaluating the psychometric properties of measures of social inclusion. PLoS ONE. 2017;12(6):e0179109.PubMedPubMedCentralCrossRefGoogle Scholar
  35. 35.
    Lysaght R, Cobigo V, Hamilton K. Inclusion as a focus of employment-related research in intellectual disability from 2000 to 2010: a scoping review. Disabil Rehabil. 2012;34(16):1339–1500.PubMedCrossRefPubMedCentralGoogle Scholar
  36. 36.
    Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. The Lancet. 2014;383(9912):166–175.CrossRefGoogle Scholar
  37. 37.
    Prinsen CAC, Mokkink LB, Bouter LM, Alonso J, Patrick DL, De Vet HCW, Terwee CB. COSMIN guideline for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27(5):1147–1157.PubMedPubMedCentralCrossRefGoogle Scholar
  38. 38.
    Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–269.PubMedPubMedCentralCrossRefGoogle Scholar
  39. 39.
    World Health Organization. WHO policy on the employment of persons with disabilities. Accessed 7 May 2019.
  40. 40.
    Terwee CB, Prinsen CAC, Chiarotto A, de Vet HC, Bouter LM, Alonso J, et al. COSMIN methodology for assessing the content validity of PROMs. User manual, 2018. Version 1.0. p. 72.Google Scholar
  41. 41.
    Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360–363.PubMedGoogle Scholar
  42. 42.
    GRADE. GRADE handbook—handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach; 2013.Google Scholar
  43. 43.
    Mokkink LB, De Vet HCW, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, Terwee CB. COSMIN risk of bias checklist for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27(5):1171–1179.PubMedCrossRefGoogle Scholar
  44. 44.
    Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, Terwee CB. COSMIN methodology for systematic reviews of patient-reported outcome measures (PROMs). User manual, 2018. Version 1.0. p. 78.Google Scholar
  45. 45.
    Mokkink LB, Terwee CB, Knol DL, Stratford PW, Alonso J, Patrick DL. The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: a clarification of its content. BMC Med Res Methodol. 2010;10(1):22.PubMedPubMedCentralCrossRefGoogle Scholar
  46. 46.
    Terwee CB, Mokkink LB, Knol DL, Ostelo R, Bouter LM, de Vet HCW. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res. 2012;21(4):651–657.PubMedCrossRefGoogle Scholar
  47. 47.
    Mokkink LB, Terwee CB, Gibbons E, Stratford PW, Alonso J, Patrick DL, et al. Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) checklist. BMC Med Res Methodol. 2010;10(1):82.PubMedPubMedCentralCrossRefGoogle Scholar
  48. 48.
    Terwee CB, Prinsen CAC, Chiarotto A, Westerman MJ, Patrick DL, Alonso J. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27(5):1159–1170.PubMedPubMedCentralCrossRefGoogle Scholar
  49. 49.
    Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174.PubMedPubMedCentralCrossRefGoogle Scholar
  50. 50.
    Altman DG. Practical statistics for medical research. Boca Raton: CRC Press; 1990.Google Scholar
  51. 51.
    Hagner DC. Breaks and birthday cakes: evaluating workplace cultures to develop natural supports for employees with disabilities. St. Augustine: Training Resource Network Inc.; 2000.Google Scholar
  52. 52.
    Mor Barak ME. Managing diversity: toward a globally inclusive workplace. Thousand Oaks: Sage Publishers; 2005.Google Scholar
  53. 53.
    Mor Barak ME, Findler L, Wind LH. Cross-cultural aspects of diversity and well-being in the workplace: an international perspective. J Soc Work Res Eval. 2003;4(2):145–169.Google Scholar
  54. 54.
    Cho S, Mor Barak ME. Understanding of diversity and inclusion in a perceived homogeneous culture: a study of organizational commitment and job performance among Korean employees. Adm Soc Work. 2008;32(4):100–126.CrossRefGoogle Scholar
  55. 55.
    Hwang J, Hopkins K. Organizational inclusion, commitment, and turnover among child welfare workers: a multilevel mediation analysis. Adm Soc Work. 2012;36(1):23–39.CrossRefGoogle Scholar
  56. 56.
    Cottrill K, Denise Lopez P, Hoffman CC. How authentic leadership and inclusion benefit organizations. Equal Divers Incl Int J. 2014;33(3):275–292.CrossRefGoogle Scholar
  57. 57.
    Goswami S, Goswami BK. Exploring the relationship between workforce diversity, inclusion and employee engagement. Drishtikon Manage J. 2018;9(1):65–89.Google Scholar
  58. 58.
    Chen C, Tang N. Does perceived inclusion matter in the workplace? J Manage Psychol. 2018;33(1):43–57.CrossRefGoogle Scholar
  59. 59.
    Ma E, Qu H, Wilson M. The affective and dispositional consequences of organizational citizenship behavior: a cross-cultural study. J Hosp Tour Res. 2016;40(4):399–43131.CrossRefGoogle Scholar
  60. 60.
    Hitlan RT, Cliffton RJ, DeSoto MC. Perceived exclusion in the workplace: the moderating effects of gender on work-related attitudes and psychological health. N Am J Psychol. 2006;8(2):217–236.Google Scholar
  61. 61.
    Hitlan RT, Noel J. The influence of workplace exclusion and personality on counterproductive work behaviours: an interactionist perspective. Eur J Work Organ Psychol. 2009;18(4):477–502.CrossRefGoogle Scholar
  62. 62.
    Hagner D, Dague B, Phillips K. Including employees with disabilities in workplace cultures: strategies and barriers. Rehabil Couns Bull. 2015;58(4):195–202.CrossRefGoogle Scholar
  63. 63.
    Jansen WS, Otten S, van der Zee KI. Being different at work: how gender dissimilarity relates to social inclusion and absenteeism. Group Process Intergroup Relat. 2017;20(6):879–893.CrossRefGoogle Scholar
  64. 64.
    Chow IH-S. Cognitive diversity and creativity in teams: the mediating roles of team learning and inclusion. Chin Manage Stud. 2018;12(2):369–383.CrossRefGoogle Scholar
  65. 65.
    Downey SN, van der Werff L, Thomas KM, Plaut VC. The role of diversity practices and inclusion in promoting trust and employee engagement. J Appl Soc Psychol. 2015;45(1):35–44.CrossRefGoogle Scholar
  66. 66.
    Fiset J, Boies K. Seeing the unseen: ostracism interventionary behaviour and its impact on employees. Eur J Work Organ Psychol. 2018;27(4):403–417.CrossRefGoogle Scholar
  67. 67.
    Jansen WS, Otten S, van der Zee KI, Jans L. Inclusion: conceptualization and measurement. Eur J Soc Psychol. 2014;44(4):370–385.CrossRefGoogle Scholar
  68. 68.
    Pearce JL, Randel AE. Expectations of organizational mobility, workplace social inclusion, and employee job performance. J Organ Behav Int J Ind Occup Organ Psychol Behav. 2004;25(1):81–988.Google Scholar
  69. 69.
    Lima JE, West GR, Winston BE, Wood JA. Measuring organizational cultural intelligence: the development and validation of a scale. Int J Cross Cult Manage. 2016;16(1):9–31.CrossRefGoogle Scholar
  70. 70.
    Ferris DL, Brown DJ, Berry JW, Lian H. The development and validation of the Workplace Ostracism Scale. J Appl Psychol. 2008;93(6):1348–1366.PubMedCrossRefPubMedCentralGoogle Scholar
  71. 71.
    Fisher RJ. Social desirability bias and the validity of indirect questioning. J Consum Res. 1993;20(2):303–315.CrossRefGoogle Scholar
  72. 72.
    Nelissen PT, Hulsheger UR, van Ruitenbeek GMC, Zijlstra FRH. Lending a helping hand at work: a multilevel investigation of prosocial motivation, inclusive climate and inclusive behavior. J Occup Rehabil. 2017;27(3):467–476.PubMedCrossRefPubMedCentralGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2020

Authors and Affiliations

  1. 1.Bloorview Research InstituteHolland Bloorview Kids Rehabilitation HospitalTorontoCanada
  2. 2.Rehabilitation Sciences InstituteUniversity of TorontoTorontoCanada
  3. 3.Department of Occupational Science and Occupational TherapyUniversity of TorontoTorontoCanada

Personalised recommendations