Estimating Group-Level Relationships: General Recommendations and Considerations for the Use of Intraclass Correlation Coefficients
- 2k Downloads
In this investigation, we argue for why and how available intraclass correlation coefficients and other types of reliability estimates can be employed as sample-based reliability estimates within primary and meta-analytic studies when relationships between group-level phenomena are of interest.
Group-level correlations and reliability estimates were obtained from 46 studies examining organizational climate–performance relationships. We illustrate how the group-level reliability estimates can be used to correct correlations for predictor and criterion unreliability. Procedures are presented for computing the sampling variances of individually corrected correlations that account for sampling error in the group-level reliability estimates.
Support was found for the conservative nature of meta-analytic parameter estimates when group-level reliability information is sample-based as opposed to assumed population values. In addition, our analyses indicated that conclusions about substantive relationships between group-level variables can change based on availability of sample-based reliabilities within both primary and meta-analytic studies.
Results from this study suggest that researchers should rely on sample-based meta-analytic procedures when examining the generalizability of group-level relationships. This study also demonstrates the importance of using all available reliability information and accounting for sampling error in the reliability estimates when conducting meta-analyses at the group level of analysis.
This study breaks ground by systematically examining the use of intraclass correlation coefficients as reliability estimates within group-level meta-analytic studies. Furthermore, illustrative analyses provide guidance to primary and meta-analytic researchers in regard to how to correct group-level correlations for unreliability in the predictor, criterion, or both whenever and in whatever proportions the artifact information is available.
KeywordsMeta-analysis Group-level data ICCs Reliability
- *Studies included in meta-analytic calculationsGoogle Scholar
- Bliese, P. D. (2000). Within group-agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349–381). San Francisco, CA: Jossey-Bass.Google Scholar
- *Borucki, C. C., & Burke, M. J. (1999). An examination of service-related antecedents to retail store performance. Journal of Organizational Behavior, 20, 943–962. doi: 10.1002/(SICI)1099-1379(199911)20:6<943::AID-JOB976>3.0.CO;2-9.CrossRefGoogle Scholar
- Burke, M. J., & Landis, R. S. (2003). Methodological and conceptual challenges in conducting and interpreting meta-analyses. In K. Murphy (Ed.), Validity generalization: A critical review (pp. 287–309). Mahwah, NJ: Erlbaum.Google Scholar
- Burke, M. J., Landis, R. J., & Burke, M. I. (2014). 80 and beyond: Recommendations for disattenuating correlations. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7, 531–535.Google Scholar
- Cronbach, L. J., Gleser, G., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.Google Scholar
- *de Jong, A., de Ruyter, K., & Lemmink, J. (2005). Service climate in self-managing teams: Mapping the linkage of team member perceptions and service performance outcomes in a business-to-business setting. Journal of Management Studies, 42, 1593–1620. doi: 10.1111/j.1467-6486.2005.00558.x.CrossRefGoogle Scholar
- *Hui, C. H., Chiu, W. C. K., Yu, P. L. H., Cheng, K., & Tse, H. (2007). The effects of service climate and the effective leadership behaviour of supervisors on frontline employee service quality: A multi-level analysis. Journal of Occupational & Organizational Psychology, 80, 151–172. doi: 10.1348/096317905X89391.CrossRefGoogle Scholar
- *King, E. B., de Chermont, K., West, M., Dawson, J. F., & Hebl, M. (2007). How innovation can alleviate negative consequences of demanding work contexts: The influence of climate for innovation on organizational outcomes. Journal of Occupational and Organizational Psychology, 80, 631–645. doi: 10.1348/096317906X171145.CrossRefGoogle Scholar
- LeBreton, J. M., Scherer, K. T., & James, L. R. (2014). Corrections for criterion reliability in validity generalization: A false prophet in a land of suspended judgment. Industrial and Organizational Psychology: Perspectives on Science and Practice, 74, 478–500.Google Scholar
- Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Menlo Park, CA: Addison-Wesley.Google Scholar
- *Luria, G. (2008). Controlling for quality: Climate, leadership, and behavior. Quality Management Journal, 15, 27–42.Google Scholar
- *Sowinski, D. R., Fortmann, K. A., & Lezotte, D. V. (2008). Climate for service and the moderating effects of climate strength on customer satisfaction, voluntary turnover, and profitability. European Journal of Work and Organizational Psychology, 17, 73–88. doi: 10.1080/13594320701473065.CrossRefGoogle Scholar
- Stanley, J. C. (1971). Reliability. In R. L. Thorndike (Ed.), Educational Measurement (2nd ed.). Washington, DC: American Council on Education.Google Scholar
- Whitman, D. S., Caleo, S., Carpenter, N. C., Horner, M. T., & Bernerth, J. B. (2012). Fairness at the collective level: A meta-analytic examination of the consequences and boundary conditions of organizational justice climate. Journal of Applied Psychology, 97(4), 776–791. doi: 10.1037/a0028021.CrossRefPubMedGoogle Scholar