Advertisement

Journal of Business and Psychology

, Volume 32, Issue 6, pp 611–626 | Cite as

Estimating Group-Level Relationships: General Recommendations and Considerations for the Use of Intraclass Correlation Coefficients

  • Maura I. BurkeEmail author
  • Ronald S. Landis
  • Michael J. Burke
Original Paper

Abstract

Purpose

In this investigation, we argue for why and how available intraclass correlation coefficients and other types of reliability estimates can be employed as sample-based reliability estimates within primary and meta-analytic studies when relationships between group-level phenomena are of interest.

Design/Methodology/Approach

Group-level correlations and reliability estimates were obtained from 46 studies examining organizational climate–performance relationships. We illustrate how the group-level reliability estimates can be used to correct correlations for predictor and criterion unreliability. Procedures are presented for computing the sampling variances of individually corrected correlations that account for sampling error in the group-level reliability estimates.

Findings

Support was found for the conservative nature of meta-analytic parameter estimates when group-level reliability information is sample-based as opposed to assumed population values. In addition, our analyses indicated that conclusions about substantive relationships between group-level variables can change based on availability of sample-based reliabilities within both primary and meta-analytic studies.

Implications

Results from this study suggest that researchers should rely on sample-based meta-analytic procedures when examining the generalizability of group-level relationships. This study also demonstrates the importance of using all available reliability information and accounting for sampling error in the reliability estimates when conducting meta-analyses at the group level of analysis.

Originality/Value

This study breaks ground by systematically examining the use of intraclass correlation coefficients as reliability estimates within group-level meta-analytic studies. Furthermore, illustrative analyses provide guidance to primary and meta-analytic researchers in regard to how to correct group-level correlations for unreliability in the predictor, criterion, or both whenever and in whatever proportions the artifact information is available.

Keywords

Meta-analysis Group-level data ICCs Reliability 

References

  1. *Studies included in meta-analytic calculationsGoogle Scholar
  2. Austin, J. T., & Villanova, P. (1992). The criterion problem: 1917–1992. Journal of Applied Psychology, 77, 836–874. doi: 10.1037/0021-9010.77.6.836.CrossRefGoogle Scholar
  3. *Baer, M., & Frese, M. (2003). Innovation is not enough: Climates for initiative and psychological safety, process innovations, and firm performance. Journal of Organizational Behavior, 24, 45–68. doi: 10.1002/job.179.CrossRefGoogle Scholar
  4. Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83, 762–765. doi: 10.1037/0033-2909.83.5.762.CrossRefGoogle Scholar
  5. Bliese, P. D. (2000). Within group-agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349–381). San Francisco, CA: Jossey-Bass.Google Scholar
  6. *Borucki, C. C., & Burke, M. J. (1999). An examination of service-related antecedents to retail store performance. Journal of Organizational Behavior, 20, 943–962. doi: 10.1002/(SICI)1099-1379(199911)20:6<943::AID-JOB976>3.0.CO;2-9.CrossRefGoogle Scholar
  7. Burke, M. J., & Landis, R. S. (2003). Methodological and conceptual challenges in conducting and interpreting meta-analyses. In K. Murphy (Ed.), Validity generalization: A critical review (pp. 287–309). Mahwah, NJ: Erlbaum.Google Scholar
  8. Burke, M. J., Landis, R. J., & Burke, M. I. (2014). 80 and beyond: Recommendations for disattenuating correlations. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7, 531–535.Google Scholar
  9. *Chen, G., Kirkman, B., Kanfer, R., Allen, D., & Rosen, B. (2007). A multilevel study of leadership, empowerment, and performance in teams. Journal of Applied Psychology, 92, 331–346. doi: 10.1037/0021-9010.92.2.331.CrossRefPubMedGoogle Scholar
  10. Christian, M. S., Bradley, J. C., Wallace, J., & Burke, M. J. (2009). Workplace safety: A meta-analysis of the roles of person and situation factors. Journal of Applied Psychology, 94(5), 1103–1127. doi: 10.1037/a0016172.CrossRefPubMedGoogle Scholar
  11. *Collins, C. J., & Smith, K. G. (2006). Knowledge exchange and combination: The role of human resource practices in the performance of high-technology firms. Academy of Management Journal, 49, 544–560. doi: 10.5465/AMJ.2006.21794671.CrossRefGoogle Scholar
  12. *Colquitt, J. A., Noe, R. A., & Jackson, C. L. (2002). Justice in teams: Antecedents and consequences of procedural justice climate. Personnel Psychology, 55, 83–109. doi: 10.1111/j.1744-6570.2002.tb00104.x.CrossRefGoogle Scholar
  13. Cronbach, L. J., Gleser, G., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.Google Scholar
  14. *Dawson, J. F., Gonzalez-Roma, V., Davis, A., & West, M. A. (2008). Organisational climate and climate strength in UK hospitals. European Journal of Work and Organizational Psychology, 17, 89–111. doi: 10.1080/13594320601046664.CrossRefGoogle Scholar
  15. *de Jong, A., de Ruyter, K., & Lemmink, J. (2005). Service climate in self-managing teams: Mapping the linkage of team member perceptions and service performance outcomes in a business-to-business setting. Journal of Management Studies, 42, 1593–1620. doi: 10.1111/j.1467-6486.2005.00558.x.CrossRefGoogle Scholar
  16. *Dietz, J., Pugh, S. D., & Wiley, J. (2004). Service climate effects on customer attitudes: An examination of boundary conditions. Academy of Management Journal, 47, 81–92. doi: 10.2307/20159561.CrossRefGoogle Scholar
  17. Ebel, R. L. (1951). Estimation of the reliability of ratings. Psychometrika, 16, 407–424. doi: 10.1007/BF02288803.CrossRefGoogle Scholar
  18. *Ehrhart, M. G. (2004). Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Personnel Psychology, 57, 61–94. doi: 10.1111/j.1744-6570.2004.tb02484.x.CrossRefGoogle Scholar
  19. *Fay, D., Luhrmann, H., & Kohl, C. (2004). Proactive climate in a post-reorganization setting: When staff compensate managers’ weakness. European Journal of Work and Organizational Psychology, 13, 241–267. doi: 10.1080/13594320444000083.CrossRefGoogle Scholar
  20. *Gelade, G. A., & Ivery, M. (2003). The impact of human resource management and work climate on organizational performance. Personnel Psychology, 56, 383–404. doi: 10.1111/j.1744-6570.2003.tb00155.x.CrossRefGoogle Scholar
  21. *Gelade, G. A., & Young, S. (2005). Test of a service profit chain model in the retail banking sector. Journal of Occupational & Organizational Psychology, 78, 1–22. doi: 10.1348/096317904X22926.CrossRefGoogle Scholar
  22. *Gil, F., Rico, R., Alcover, C. M., & Barrasa, A. (2005). Change-oriented leadership, satisfaction and performance in work groups. Journal of Managerial Psychology, 20, 312–328. doi: 10.1108/02683940510589073.CrossRefGoogle Scholar
  23. *Gonzalez-Roma, V., Fortes-Ferreira, L., & Peiro, J. (2009). Team climate, climate strength and team performance. A longitudinal study. Journal of Occupational and Organizational Psychology, 82, 511–536. doi: 10.1348/096317908X370025.CrossRefGoogle Scholar
  24. Harter, J. K., Schmidt, F. L., Asplund, J. W., Killham, E. A., & Agrawal, S. (2010). Causal impact of employee work perceptions on the bottom line of organizations. Perspectives on Psychological Science, 5(4), 378–389. doi: 10.1177/1745691610374589.CrossRefPubMedGoogle Scholar
  25. Harter, J. K., Schmidt, F. L., & Hayes, T. L. (2002). Business-unit-level relationship between employee satisfaction, employee engagement, and business outcomes: A meta-analysis. Journal of Applied Psychology, 87(2), 268–279. doi: 10.1037/0021-9010.87.2.268.CrossRefPubMedGoogle Scholar
  26. *Hofmann, D. A., & Mark, B. (2006). An investigation of the relationship between safety climate and medication errors as well as other nurse and patient outcomes. Personnel Psychology, 59, 847–869. doi: 10.1111/j.1744-6570.2006.00056.x.CrossRefGoogle Scholar
  27. *Hofmann, D. A., & Stetzer, A. (1996). A cross-level investigation of factors influencing unsafe behaviors and accidents. Personnel Psychology, 49, 307–339. doi: 10.1111/j.1744-6570.1996.tb01802.x.CrossRefGoogle Scholar
  28. Hong, Y., Liao, H., Hu, J., & Jiang, K. (2013). Missing link in the service profit chain: A meta-analytic review of the antecedents, consequences, and moderators of service climate. Journal of Applied Psychology, 98(2), 237–267. doi: 10.1037/a0031666.CrossRefPubMedGoogle Scholar
  29. *Hui, C. H., Chiu, W. C. K., Yu, P. L. H., Cheng, K., & Tse, H. (2007). The effects of service climate and the effective leadership behaviour of supervisors on frontline employee service quality: A multi-level analysis. Journal of Occupational & Organizational Psychology, 80, 151–172. doi: 10.1348/096317905X89391.CrossRefGoogle Scholar
  30. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
  31. *Katz-Navon, T., Naveh, E., & Stern, Z. (2005). Safety climate in healthcare organizations: A multidimensional approach. Academy of Management Journal, 48, 1075–1089. doi: 10.5465/AMJ.2005.19573110.CrossRefGoogle Scholar
  32. *King, E. B., de Chermont, K., West, M., Dawson, J. F., & Hebl, M. (2007). How innovation can alleviate negative consequences of demanding work contexts: The influence of climate for innovation on organizational outcomes. Journal of Occupational and Organizational Psychology, 80, 631–645. doi: 10.1348/096317906X171145.CrossRefGoogle Scholar
  33. Lahey, M., Downey, R. G., & Saal, F. E. (1983). Intraclass correlations: There’s more there than meets the eye. Psychological Bulletin, 93(3), 586–595. doi: 10.1037/0033-2909.93.3.586.CrossRefGoogle Scholar
  34. LeBreton, J. M., Scherer, K. T., & James, L. R. (2014). Corrections for criterion reliability in validity generalization: A false prophet in a land of suspended judgment. Industrial and Organizational Psychology: Perspectives on Science and Practice, 74, 478–500.Google Scholar
  35. *Liao, H., & Chuang, A. (2004). A multilevel investigation of factors influencing employee service performance and customer outcomes. Academy of Management Journal, 47, 41–58. doi: 10.2307/20159559.CrossRefGoogle Scholar
  36. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Menlo Park, CA: Addison-Wesley.Google Scholar
  37. *Luria, G. (2008). Controlling for quality: Climate, leadership, and behavior. Quality Management Journal, 15, 27–42.Google Scholar
  38. *Luria, G., & Yagil, D. (2008). Procedural justice, ethical climate and service outcomes in restaurants. International Journal of Hospitality Management, 27, 276–283. doi: 10.1016/j.ijhm.2007.07.022.CrossRefGoogle Scholar
  39. *MacCormick, J. S., & Parker, S. K. (2010). A multiple climates approach to understanding business unit effectiveness. Human Relations, 63, 1171–1806. doi: 10.1177/0018726710365090.Google Scholar
  40. *Mayer, D. M., Kuenzi, M., & Greenbaum, R. L. (2010). Examining the link between ethical leadership and employee misconduct: The mediating role of ethical climate. Journal of Business Ethics, 95, 7–16. doi: 10.1007/s10551-0110-794-0.CrossRefGoogle Scholar
  41. *Maynard, M. T., Mathieu, J. E., Marsh, W. M., & Ruddy, T. M. (2007). A multilevel investigation of the influences of employees’ resistance to empowerment. Human Performance, 20, 147–171. doi: 10.1080/08959280701332885.CrossRefGoogle Scholar
  42. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46. doi: 10.1037/1082-989X.1.1.30.CrossRefGoogle Scholar
  43. *McKay, P. F., Avery, D. R., & Morris, M. A. (2009). A tale of two climates: Diversity climate from subordinates’ and managers’ perspectives and their role in store unit sales performance. Personnel Psychology, 62, 767–791. doi: 10.1111/j.1744-6570.2009.01157.x.CrossRefGoogle Scholar
  44. *Naumann, S., & Bennett, N. (2002). The effects of procedural justice climate on work group performance. Small Group Research, 33, 361–377. doi: 10.1177/10496402033003004.CrossRefGoogle Scholar
  45. *Naveh, E., Katz-Navon, T., & Stern, Z. (2005). Treatment errors in healthcare: A safety climate approach. Management Science, 51, 948–960. doi: 10.1287/mnsc.1050.0372.CrossRefGoogle Scholar
  46. *Neal, A., & Griffin, M. A. (2006). A study of the lagged relationships among safety climate, safety motivation, safety behavior, and accidents at the individual and group levels. Journal of Applied Psychology, 91, 946–953. doi: 10.1037/0021-9010.91.4.946.CrossRefPubMedGoogle Scholar
  47. *Ostroff, C., Kinicki, A., & Clark, M. (2002). Substantive and operational issues of response bias across levels of analysis: An example of climate-satisfaction relationships. Journal of Applied Psychology, 87, 355–368. doi: 10.1037/0021-9010.87.2.355.CrossRefPubMedGoogle Scholar
  48. *Pirola-Merlo, A., Hartel, C., Mann, L., & Hirst, G. (2002). How leaders influence the impact of affective events on team climate and performance in R&D teams. Leadership Quarterly, 13, 561–581. doi: 10.1016/S1048-9843(02)00144-3.CrossRefGoogle Scholar
  49. Raju, N. S., & Brand, P. A. (2003). Determining the significance of correlations corrected for unreliability and range restriction. Applied Psychological Measurement, 27, 52–71. doi: 10.1177/0146621602239476.CrossRefGoogle Scholar
  50. Raju, N. S., Burke, M. J., Normand, J., & Langlois, G. M. (1991). A new meta-analytic approach. Journal of Applied Psychology, 76, 432–446. doi: 10.1037/0021-9010.76.3.432.CrossRefGoogle Scholar
  51. Raju, N. S., Pappas, S., & Williams, C. P. (1989). An empirical Monte Carlo test of the accuracy of the correlation, covariance, and regression slope models for assessing validity generalization. Journal of Applied Psychology, 74, 901–911. doi: 10.1037/0021-9010.74.6.901.CrossRefGoogle Scholar
  52. *Richardson, H. A., & Vandenberg, R. J. (2005). Integrating managerial perceptions and transformational leadership into a work-unit level model of employee involvement. Journal of Organizational Behavior, 26, 561–589. doi: 10.1002/job.329.CrossRefGoogle Scholar
  53. *Salanova, M., Agut, S., & Peiro, J. M. (2005). Linking organizational resources and work engagement to employee performance and customer loyalty: The mediation of service climate. Journal of Applied Psychology, 90, 1217–1227. doi: 10.1037/0021-9010.90.6.1217.CrossRefPubMedGoogle Scholar
  54. *Schmit, M. J., & Allscheid, S. P. (1995). Employee attitudes and customer satisfaction: Making theoretical and empirical connections. Personnel Psychology, 48, 521–536. doi: 10.1111/j.1744-6570.1995.tb01768.x.CrossRefGoogle Scholar
  55. *Schneider, B., Ehrhart, M. G., Mayer, D. M., Saltz, J. L., & Niles-Jolly, K. (2005). Understanding organization-customer links in service settings. Academy of Management Journal, 48, 1017–1032. doi: 10.5465/AMJ.2005.19573107.CrossRefGoogle Scholar
  56. *Schneider, B., White, S. S., & Paul, M. C. (1998). Linking service climate and customer perceptions of service quality: Test of a causal model. Journal of Applied Psychology, 83, 150–163. doi: 10.1037/0021-9010.83.2.150.CrossRefPubMedGoogle Scholar
  57. *Schulte, M., Ostroff, C., Shmulyian, S., & Kinicki, A. (2009). Organizational climate configurations: Relationships to collective attitudes, customer satisfaction, and financial performance. Journal of Applied Psychology, 3, 618–634. doi: 10.1037/a0014365.CrossRefGoogle Scholar
  58. *Seibert, S. E., Silver, S. R., & Randolph, W. A. (2004). Taking empowerment to the next level: A multiple-level model of empowerment, performance, and satisfaction. Academy of Management Journal, 47, 332–349. doi: 10.2307/20159585.CrossRefGoogle Scholar
  59. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. doi: 10.1037/0033-2909.86.2.420.CrossRefPubMedGoogle Scholar
  60. *Simons, T., & Roberson, Q. (2003). Why managers should care about fairness: The effects of aggregate justice perceptions on organizational outcomes. Journal of Applied Psychology, 88, 432–443. doi: 10.1037/0021-9010.88.3.432.CrossRefPubMedGoogle Scholar
  61. *Smith, K. G., Collins, C. J., & Clark, K. D. (2005). Existing knowledge, knowledge creation capability, and the rate of new product introduction in high-technology firms. Academy of Management Journal, 48, 346–357. doi: 10.5465/AMJ.2005.16928421.CrossRefGoogle Scholar
  62. *Sowinski, D. R., Fortmann, K. A., & Lezotte, D. V. (2008). Climate for service and the moderating effects of climate strength on customer satisfaction, voluntary turnover, and profitability. European Journal of Work and Organizational Psychology, 17, 73–88. doi: 10.1080/13594320701473065.CrossRefGoogle Scholar
  63. Stanley, J. C. (1971). Reliability. In R. L. Thorndike (Ed.), Educational Measurement (2nd ed.). Washington, DC: American Council on Education.Google Scholar
  64. *Stetzer, A., Morgeson, F., & Anderson, E. (1997). Organizational climate and ineffectiveness: Evidence from 25 outdoor workcrew divisions. Journal of Quality Management, 2, 251–265. doi: 10.1016/S1084-8568(97)90006-7.CrossRefGoogle Scholar
  65. Thoreson, C. J., Kaplan, S. A., Barsky, A. P., Warren, C. R., & de Chermont, K. (2003). The affective underpinnings of job perceptions and attitudes: A meta-analytic review and integration. Psychological Bulletin, 129, 914–945. doi: 10.1037/0033-2909.129.6.914.CrossRefGoogle Scholar
  66. *Towler, A., Lezotte, D. V., & Burke, M. J. (2011). An examination of the service climate-firm performance chain: The role of customer retention. Human Resource Management, 50, 391–406. doi: 10.1002/hrm.20422.CrossRefGoogle Scholar
  67. *Wagner, S. H., Parker, C. P., & Christiansen, N. D. (2003). Employees that think and act like owners: Effects of ownerships beliefs and behaviors on organizational effectiveness. Personnel Psychology, 56, 847–871. doi: 10.1111/j.1744-6570.2003.tb00242.x.CrossRefGoogle Scholar
  68. Wallace, J. C., Edwards, B. D., Paul, J., Burke, M. J., Christian, M., & Eissa, G. (2016). Change the referent? A meta-analytic investigation of direct and referent-shift consensus models for organizational climate. Journal of Management, 42, 838–861.CrossRefGoogle Scholar
  69. *Wallace, C., & Chen, G. (2006). A multilevel integration of personality, climate, self-regulation, and performance. Personnel Psychology, 59, 529–557. doi: 10.1111/j.1744-6570.2006.00046.x.CrossRefGoogle Scholar
  70. Whitman, D. S., Caleo, S., Carpenter, N. C., Horner, M. T., & Bernerth, J. B. (2012). Fairness at the collective level: A meta-analytic examination of the consequences and boundary conditions of organizational justice climate. Journal of Applied Psychology, 97(4), 776–791. doi: 10.1037/a0028021.CrossRefPubMedGoogle Scholar
  71. Whitman, D. S., Van Rooy, D. L., & Viswesvaran, C. (2010). Satisfaction, citizenship behaviors, and performance in work units: A meta-analysis of collective construct relations. Personnel Psychology, 63(1), 41–81. doi: 10.1111/j.1744-6570.2009.01162.x.CrossRefGoogle Scholar
  72. *Zohar, D., & Luria, G. (2005). A multilevel model of safety climate: Cross level relationships between organization and group-level climates. Journal of Applied Psychology, 90, 616–628. doi: 10.1037/0021-9010.90.4.616.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Maura I. Burke
    • 1
    Email author
  • Ronald S. Landis
    • 1
  • Michael J. Burke
    • 2
  1. 1.Department of PsychologyIllinois Institute of TechnologyChicagoUSA
  2. 2.A.B. Freeman School of BusinessTulane UniversityNew OrleansUSA

Personalised recommendations