Skip to main content

Estimating Group-Level Relationships: General Recommendations and Considerations for the Use of Intraclass Correlation Coefficients

Abstract

Purpose

In this investigation, we argue for why and how available intraclass correlation coefficients and other types of reliability estimates can be employed as sample-based reliability estimates within primary and meta-analytic studies when relationships between group-level phenomena are of interest.

Design/Methodology/Approach

Group-level correlations and reliability estimates were obtained from 46 studies examining organizational climate–performance relationships. We illustrate how the group-level reliability estimates can be used to correct correlations for predictor and criterion unreliability. Procedures are presented for computing the sampling variances of individually corrected correlations that account for sampling error in the group-level reliability estimates.

Findings

Support was found for the conservative nature of meta-analytic parameter estimates when group-level reliability information is sample-based as opposed to assumed population values. In addition, our analyses indicated that conclusions about substantive relationships between group-level variables can change based on availability of sample-based reliabilities within both primary and meta-analytic studies.

Implications

Results from this study suggest that researchers should rely on sample-based meta-analytic procedures when examining the generalizability of group-level relationships. This study also demonstrates the importance of using all available reliability information and accounting for sampling error in the reliability estimates when conducting meta-analyses at the group level of analysis.

Originality/Value

This study breaks ground by systematically examining the use of intraclass correlation coefficients as reliability estimates within group-level meta-analytic studies. Furthermore, illustrative analyses provide guidance to primary and meta-analytic researchers in regard to how to correct group-level correlations for unreliability in the predictor, criterion, or both whenever and in whatever proportions the artifact information is available.

This is a preview of subscription content, access via your institution.

Notes

  1. Equations used have been reported elsewhere as noted. Numbering of the equations in the current paper is for ease of presentation only.

  2. The RBNL meta-analytic procedures for correcting individual correlations depart from those proposed by Hunter and Schmidt (2004) by accounting not only for sampling error due to N, but additionally take into account sampling error in the reliability estimates. As discussed in Raju et al. (1991), Hunter and Schmidt’s (2004) meta-analytic procedures for individually corrected correlations deal with the special case in which reliabilities of predictor and criterion are assumed to be fixed. The reader is referred to Raju et al. (1991) for a detailed description concerning the advantages of employing sample-based meta-analytic procedures over alternative methods.

  3. One study was excluded from the analyses due to a large sample size and the direction of the effect. Results of these analyses with the study included are available upon request from the author.

  4. Supplemental meta-analyses were also conducted with a 75 % reduction in available reliability from the original amount and are available upon request from the first author. Most notably, when sample-based reliabilities (and their associated sampling variances) are artificially extracted we see a marked increase in \({\text{SE}}_{{M_{\rho } }}\) and a substantially larger magnitude of M ρ . Furthermore, the range of confidence moves farther away from original surrounding calculations when the data naturally occurred.

References

  • *Studies included in meta-analytic calculations

  • Austin, J. T., & Villanova, P. (1992). The criterion problem: 1917–1992. Journal of Applied Psychology, 77, 836–874. doi:10.1037/0021-9010.77.6.836.

    Article  Google Scholar 

  • *Baer, M., & Frese, M. (2003). Innovation is not enough: Climates for initiative and psychological safety, process innovations, and firm performance. Journal of Organizational Behavior, 24, 45–68. doi:10.1002/job.179.

    Article  Google Scholar 

  • Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83, 762–765. doi:10.1037/0033-2909.83.5.762.

    Article  Google Scholar 

  • Bliese, P. D. (2000). Within group-agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349–381). San Francisco, CA: Jossey-Bass.

    Google Scholar 

  • *Borucki, C. C., & Burke, M. J. (1999). An examination of service-related antecedents to retail store performance. Journal of Organizational Behavior, 20, 943–962. doi:10.1002/(SICI)1099-1379(199911)20:6<943::AID-JOB976>3.0.CO;2-9.

    Article  Google Scholar 

  • Burke, M. J., & Landis, R. S. (2003). Methodological and conceptual challenges in conducting and interpreting meta-analyses. In K. Murphy (Ed.), Validity generalization: A critical review (pp. 287–309). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Burke, M. J., Landis, R. J., & Burke, M. I. (2014). 80 and beyond: Recommendations for disattenuating correlations. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7, 531–535.

    Google Scholar 

  • *Chen, G., Kirkman, B., Kanfer, R., Allen, D., & Rosen, B. (2007). A multilevel study of leadership, empowerment, and performance in teams. Journal of Applied Psychology, 92, 331–346. doi:10.1037/0021-9010.92.2.331.

    Article  PubMed  Google Scholar 

  • Christian, M. S., Bradley, J. C., Wallace, J., & Burke, M. J. (2009). Workplace safety: A meta-analysis of the roles of person and situation factors. Journal of Applied Psychology, 94(5), 1103–1127. doi:10.1037/a0016172.

    Article  PubMed  Google Scholar 

  • *Collins, C. J., & Smith, K. G. (2006). Knowledge exchange and combination: The role of human resource practices in the performance of high-technology firms. Academy of Management Journal, 49, 544–560. doi:10.5465/AMJ.2006.21794671.

    Article  Google Scholar 

  • *Colquitt, J. A., Noe, R. A., & Jackson, C. L. (2002). Justice in teams: Antecedents and consequences of procedural justice climate. Personnel Psychology, 55, 83–109. doi:10.1111/j.1744-6570.2002.tb00104.x.

    Article  Google Scholar 

  • Cronbach, L. J., Gleser, G., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.

    Google Scholar 

  • *Dawson, J. F., Gonzalez-Roma, V., Davis, A., & West, M. A. (2008). Organisational climate and climate strength in UK hospitals. European Journal of Work and Organizational Psychology, 17, 89–111. doi:10.1080/13594320601046664.

    Article  Google Scholar 

  • *de Jong, A., de Ruyter, K., & Lemmink, J. (2005). Service climate in self-managing teams: Mapping the linkage of team member perceptions and service performance outcomes in a business-to-business setting. Journal of Management Studies, 42, 1593–1620. doi:10.1111/j.1467-6486.2005.00558.x.

    Article  Google Scholar 

  • *Dietz, J., Pugh, S. D., & Wiley, J. (2004). Service climate effects on customer attitudes: An examination of boundary conditions. Academy of Management Journal, 47, 81–92. doi:10.2307/20159561.

    Article  Google Scholar 

  • Ebel, R. L. (1951). Estimation of the reliability of ratings. Psychometrika, 16, 407–424. doi:10.1007/BF02288803.

    Article  Google Scholar 

  • *Ehrhart, M. G. (2004). Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Personnel Psychology, 57, 61–94. doi:10.1111/j.1744-6570.2004.tb02484.x.

    Article  Google Scholar 

  • *Fay, D., Luhrmann, H., & Kohl, C. (2004). Proactive climate in a post-reorganization setting: When staff compensate managers’ weakness. European Journal of Work and Organizational Psychology, 13, 241–267. doi:10.1080/13594320444000083.

    Article  Google Scholar 

  • *Gelade, G. A., & Ivery, M. (2003). The impact of human resource management and work climate on organizational performance. Personnel Psychology, 56, 383–404. doi:10.1111/j.1744-6570.2003.tb00155.x.

    Article  Google Scholar 

  • *Gelade, G. A., & Young, S. (2005). Test of a service profit chain model in the retail banking sector. Journal of Occupational & Organizational Psychology, 78, 1–22. doi:10.1348/096317904X22926.

    Article  Google Scholar 

  • *Gil, F., Rico, R., Alcover, C. M., & Barrasa, A. (2005). Change-oriented leadership, satisfaction and performance in work groups. Journal of Managerial Psychology, 20, 312–328. doi:10.1108/02683940510589073.

    Article  Google Scholar 

  • *Gonzalez-Roma, V., Fortes-Ferreira, L., & Peiro, J. (2009). Team climate, climate strength and team performance. A longitudinal study. Journal of Occupational and Organizational Psychology, 82, 511–536. doi:10.1348/096317908X370025.

    Article  Google Scholar 

  • Harter, J. K., Schmidt, F. L., Asplund, J. W., Killham, E. A., & Agrawal, S. (2010). Causal impact of employee work perceptions on the bottom line of organizations. Perspectives on Psychological Science, 5(4), 378–389. doi:10.1177/1745691610374589.

    Article  PubMed  Google Scholar 

  • Harter, J. K., Schmidt, F. L., & Hayes, T. L. (2002). Business-unit-level relationship between employee satisfaction, employee engagement, and business outcomes: A meta-analysis. Journal of Applied Psychology, 87(2), 268–279. doi:10.1037/0021-9010.87.2.268.

    Article  PubMed  Google Scholar 

  • *Hofmann, D. A., & Mark, B. (2006). An investigation of the relationship between safety climate and medication errors as well as other nurse and patient outcomes. Personnel Psychology, 59, 847–869. doi:10.1111/j.1744-6570.2006.00056.x.

    Article  Google Scholar 

  • *Hofmann, D. A., & Stetzer, A. (1996). A cross-level investigation of factors influencing unsafe behaviors and accidents. Personnel Psychology, 49, 307–339. doi:10.1111/j.1744-6570.1996.tb01802.x.

    Article  Google Scholar 

  • Hong, Y., Liao, H., Hu, J., & Jiang, K. (2013). Missing link in the service profit chain: A meta-analytic review of the antecedents, consequences, and moderators of service climate. Journal of Applied Psychology, 98(2), 237–267. doi:10.1037/a0031666.

    Article  PubMed  Google Scholar 

  • *Hui, C. H., Chiu, W. C. K., Yu, P. L. H., Cheng, K., & Tse, H. (2007). The effects of service climate and the effective leadership behaviour of supervisors on frontline employee service quality: A multi-level analysis. Journal of Occupational & Organizational Psychology, 80, 151–172. doi:10.1348/096317905X89391.

    Article  Google Scholar 

  • Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.

    Book  Google Scholar 

  • *Katz-Navon, T., Naveh, E., & Stern, Z. (2005). Safety climate in healthcare organizations: A multidimensional approach. Academy of Management Journal, 48, 1075–1089. doi:10.5465/AMJ.2005.19573110.

    Article  Google Scholar 

  • *King, E. B., de Chermont, K., West, M., Dawson, J. F., & Hebl, M. (2007). How innovation can alleviate negative consequences of demanding work contexts: The influence of climate for innovation on organizational outcomes. Journal of Occupational and Organizational Psychology, 80, 631–645. doi:10.1348/096317906X171145.

    Article  Google Scholar 

  • Lahey, M., Downey, R. G., & Saal, F. E. (1983). Intraclass correlations: There’s more there than meets the eye. Psychological Bulletin, 93(3), 586–595. doi:10.1037/0033-2909.93.3.586.

    Article  Google Scholar 

  • LeBreton, J. M., Scherer, K. T., & James, L. R. (2014). Corrections for criterion reliability in validity generalization: A false prophet in a land of suspended judgment. Industrial and Organizational Psychology: Perspectives on Science and Practice, 74, 478–500.

    Google Scholar 

  • *Liao, H., & Chuang, A. (2004). A multilevel investigation of factors influencing employee service performance and customer outcomes. Academy of Management Journal, 47, 41–58. doi:10.2307/20159559.

    Article  Google Scholar 

  • Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Menlo Park, CA: Addison-Wesley.

    Google Scholar 

  • *Luria, G. (2008). Controlling for quality: Climate, leadership, and behavior. Quality Management Journal, 15, 27–42.

    Google Scholar 

  • *Luria, G., & Yagil, D. (2008). Procedural justice, ethical climate and service outcomes in restaurants. International Journal of Hospitality Management, 27, 276–283. doi:10.1016/j.ijhm.2007.07.022.

    Article  Google Scholar 

  • *MacCormick, J. S., & Parker, S. K. (2010). A multiple climates approach to understanding business unit effectiveness. Human Relations, 63, 1171–1806. doi:10.1177/0018726710365090.

    Google Scholar 

  • *Mayer, D. M., Kuenzi, M., & Greenbaum, R. L. (2010). Examining the link between ethical leadership and employee misconduct: The mediating role of ethical climate. Journal of Business Ethics, 95, 7–16. doi:10.1007/s10551-0110-794-0.

    Article  Google Scholar 

  • *Maynard, M. T., Mathieu, J. E., Marsh, W. M., & Ruddy, T. M. (2007). A multilevel investigation of the influences of employees’ resistance to empowerment. Human Performance, 20, 147–171. doi:10.1080/08959280701332885.

    Article  Google Scholar 

  • McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46. doi:10.1037/1082-989X.1.1.30.

    Article  Google Scholar 

  • *McKay, P. F., Avery, D. R., & Morris, M. A. (2009). A tale of two climates: Diversity climate from subordinates’ and managers’ perspectives and their role in store unit sales performance. Personnel Psychology, 62, 767–791. doi:10.1111/j.1744-6570.2009.01157.x.

    Article  Google Scholar 

  • *Naumann, S., & Bennett, N. (2002). The effects of procedural justice climate on work group performance. Small Group Research, 33, 361–377. doi:10.1177/10496402033003004.

    Article  Google Scholar 

  • *Naveh, E., Katz-Navon, T., & Stern, Z. (2005). Treatment errors in healthcare: A safety climate approach. Management Science, 51, 948–960. doi:10.1287/mnsc.1050.0372.

    Article  Google Scholar 

  • *Neal, A., & Griffin, M. A. (2006). A study of the lagged relationships among safety climate, safety motivation, safety behavior, and accidents at the individual and group levels. Journal of Applied Psychology, 91, 946–953. doi:10.1037/0021-9010.91.4.946.

    Article  PubMed  Google Scholar 

  • *Ostroff, C., Kinicki, A., & Clark, M. (2002). Substantive and operational issues of response bias across levels of analysis: An example of climate-satisfaction relationships. Journal of Applied Psychology, 87, 355–368. doi:10.1037/0021-9010.87.2.355.

    Article  PubMed  Google Scholar 

  • *Pirola-Merlo, A., Hartel, C., Mann, L., & Hirst, G. (2002). How leaders influence the impact of affective events on team climate and performance in R&D teams. Leadership Quarterly, 13, 561–581. doi:10.1016/S1048-9843(02)00144-3.

    Article  Google Scholar 

  • Raju, N. S., & Brand, P. A. (2003). Determining the significance of correlations corrected for unreliability and range restriction. Applied Psychological Measurement, 27, 52–71. doi:10.1177/0146621602239476.

    Article  Google Scholar 

  • Raju, N. S., Burke, M. J., Normand, J., & Langlois, G. M. (1991). A new meta-analytic approach. Journal of Applied Psychology, 76, 432–446. doi:10.1037/0021-9010.76.3.432.

    Article  Google Scholar 

  • Raju, N. S., Pappas, S., & Williams, C. P. (1989). An empirical Monte Carlo test of the accuracy of the correlation, covariance, and regression slope models for assessing validity generalization. Journal of Applied Psychology, 74, 901–911. doi:10.1037/0021-9010.74.6.901.

    Article  Google Scholar 

  • *Richardson, H. A., & Vandenberg, R. J. (2005). Integrating managerial perceptions and transformational leadership into a work-unit level model of employee involvement. Journal of Organizational Behavior, 26, 561–589. doi:10.1002/job.329.

    Article  Google Scholar 

  • *Salanova, M., Agut, S., & Peiro, J. M. (2005). Linking organizational resources and work engagement to employee performance and customer loyalty: The mediation of service climate. Journal of Applied Psychology, 90, 1217–1227. doi:10.1037/0021-9010.90.6.1217.

    Article  PubMed  Google Scholar 

  • *Schmit, M. J., & Allscheid, S. P. (1995). Employee attitudes and customer satisfaction: Making theoretical and empirical connections. Personnel Psychology, 48, 521–536. doi:10.1111/j.1744-6570.1995.tb01768.x.

    Article  Google Scholar 

  • *Schneider, B., Ehrhart, M. G., Mayer, D. M., Saltz, J. L., & Niles-Jolly, K. (2005). Understanding organization-customer links in service settings. Academy of Management Journal, 48, 1017–1032. doi:10.5465/AMJ.2005.19573107.

    Article  Google Scholar 

  • *Schneider, B., White, S. S., & Paul, M. C. (1998). Linking service climate and customer perceptions of service quality: Test of a causal model. Journal of Applied Psychology, 83, 150–163. doi:10.1037/0021-9010.83.2.150.

    Article  PubMed  Google Scholar 

  • *Schulte, M., Ostroff, C., Shmulyian, S., & Kinicki, A. (2009). Organizational climate configurations: Relationships to collective attitudes, customer satisfaction, and financial performance. Journal of Applied Psychology, 3, 618–634. doi:10.1037/a0014365.

    Article  Google Scholar 

  • *Seibert, S. E., Silver, S. R., & Randolph, W. A. (2004). Taking empowerment to the next level: A multiple-level model of empowerment, performance, and satisfaction. Academy of Management Journal, 47, 332–349. doi:10.2307/20159585.

    Article  Google Scholar 

  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. doi:10.1037/0033-2909.86.2.420.

    Article  PubMed  Google Scholar 

  • *Simons, T., & Roberson, Q. (2003). Why managers should care about fairness: The effects of aggregate justice perceptions on organizational outcomes. Journal of Applied Psychology, 88, 432–443. doi:10.1037/0021-9010.88.3.432.

    Article  PubMed  Google Scholar 

  • *Smith, K. G., Collins, C. J., & Clark, K. D. (2005). Existing knowledge, knowledge creation capability, and the rate of new product introduction in high-technology firms. Academy of Management Journal, 48, 346–357. doi:10.5465/AMJ.2005.16928421.

    Article  Google Scholar 

  • *Sowinski, D. R., Fortmann, K. A., & Lezotte, D. V. (2008). Climate for service and the moderating effects of climate strength on customer satisfaction, voluntary turnover, and profitability. European Journal of Work and Organizational Psychology, 17, 73–88. doi:10.1080/13594320701473065.

    Article  Google Scholar 

  • Stanley, J. C. (1971). Reliability. In R. L. Thorndike (Ed.), Educational Measurement (2nd ed.). Washington, DC: American Council on Education.

  • *Stetzer, A., Morgeson, F., & Anderson, E. (1997). Organizational climate and ineffectiveness: Evidence from 25 outdoor workcrew divisions. Journal of Quality Management, 2, 251–265. doi:10.1016/S1084-8568(97)90006-7.

    Article  Google Scholar 

  • Thoreson, C. J., Kaplan, S. A., Barsky, A. P., Warren, C. R., & de Chermont, K. (2003). The affective underpinnings of job perceptions and attitudes: A meta-analytic review and integration. Psychological Bulletin, 129, 914–945. doi:10.1037/0033-2909.129.6.914.

    Article  Google Scholar 

  • *Towler, A., Lezotte, D. V., & Burke, M. J. (2011). An examination of the service climate-firm performance chain: The role of customer retention. Human Resource Management, 50, 391–406. doi:10.1002/hrm.20422.

    Article  Google Scholar 

  • *Wagner, S. H., Parker, C. P., & Christiansen, N. D. (2003). Employees that think and act like owners: Effects of ownerships beliefs and behaviors on organizational effectiveness. Personnel Psychology, 56, 847–871. doi:10.1111/j.1744-6570.2003.tb00242.x.

    Article  Google Scholar 

  • Wallace, J. C., Edwards, B. D., Paul, J., Burke, M. J., Christian, M., & Eissa, G. (2016). Change the referent? A meta-analytic investigation of direct and referent-shift consensus models for organizational climate. Journal of Management, 42, 838–861.

    Article  Google Scholar 

  • *Wallace, C., & Chen, G. (2006). A multilevel integration of personality, climate, self-regulation, and performance. Personnel Psychology, 59, 529–557. doi:10.1111/j.1744-6570.2006.00046.x.

    Article  Google Scholar 

  • Whitman, D. S., Caleo, S., Carpenter, N. C., Horner, M. T., & Bernerth, J. B. (2012). Fairness at the collective level: A meta-analytic examination of the consequences and boundary conditions of organizational justice climate. Journal of Applied Psychology, 97(4), 776–791. doi:10.1037/a0028021.

    Article  PubMed  Google Scholar 

  • Whitman, D. S., Van Rooy, D. L., & Viswesvaran, C. (2010). Satisfaction, citizenship behaviors, and performance in work units: A meta-analysis of collective construct relations. Personnel Psychology, 63(1), 41–81. doi:10.1111/j.1744-6570.2009.01162.x.

    Article  Google Scholar 

  • *Zohar, D., & Luria, G. (2005). A multilevel model of safety climate: Cross level relationships between organization and group-level climates. Journal of Applied Psychology, 90, 616–628. doi:10.1037/0021-9010.90.4.616.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Maura I. Burke.

Additional information

The RBNL meta-analytic program is available upon request from the third author at mburke1@tulane.edu.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Burke, M.I., Landis, R.S. & Burke, M.J. Estimating Group-Level Relationships: General Recommendations and Considerations for the Use of Intraclass Correlation Coefficients. J Bus Psychol 32, 611–626 (2017). https://doi.org/10.1007/s10869-016-9464-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-016-9464-7

Keywords

  • Meta-analysis
  • Group-level data
  • ICCs
  • Reliability