Skip to main content
SpringerLink
Log in

Back to Basics with Mixed-Effects Models: Nine Take-Away Points

  • Original Paper
  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

Purpose

Multilevel mixed effects models are widely used in organizational behavior and organizational psychology to test and advance theory. At times, however, the complexity of the models leads researchers to draw erroneous inferences or otherwise use the models in less than optimal ways. We present nine take-away points intended to enhance the theoretical precision and utility of the models.

Approach

We demonstrate our points using two types of simulated data: one in which group membership is irrelevant, and the other in which relationships exist only because of group membership. We then demonstrate that the effects we observe in simulated data replicate in organizational data.

Findings

Little that we address will be new to methodology experts; nonetheless, we draw together a variety of points that we believe will help advance both theory and analytic rigor in multilevel analyses.

Implications

We make two points that run somewhat counter to conventional norms. First, we argue that mixed-effects models are appropriate even when ICC(1) values associated with the outcome data are small and non-significant. Second, we show that high ICC(2) values are not a prerequisite for detecting emergent multilevel relationships.

Originality/Value

The article is designed to be a resource for researchers who are learning about and applying mixed-effects (i.e., multilevel) models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. We are indebted to the late Larry James who (to our knowledge) was the originator of this example and used it when discussing these ideas in conferences.

  2. There are two ways to calculate ICC(1)—using ANOVA or mixed-effects models (Bartko 1976; Bliese 2000). Using the ANOVA method here would produce a slightly negative ICC(1), which is sometimes truncated to zero in reporting. The differences between the two methods are generally not substantial, but the decision point is worth noting.

  3. Econometricians have noted that random effect models can yield level 1 effects that are biased because they “inherit” group-level effects. Including group means as a predictor in random effect models removes this bias and produces coefficients for x that are comparable to fixed-effects models (see Raudenbush 2009).

  4. The estimate of 0.283 is identical to the values reported in Tables 3 and 4. Table 3 therefore shows that one can remove any group effects from X by group-mean centering or by including the group mean of X into the model as noted in footnote 3.

  5. The appendix provides code to create an X2 variable that randomly sorts the X variable on a group-by-group basis and then re-estimates the components of the covariance theorem. As anticipated, aligning Y variables to different X responses within a group has no meaningful impact on the results.

  6. Interested readers can execute the final simulation in the appendix to see how a group-mean interaction can be detected in a cross-level interaction model and then how group-mean centering corrects the problem.

References

  • Aguinis, H., & Molina-Azorin, J. F. (2015). Using multilevel modeling and mixed methods to make theoretical progress in microfoundations for strategy research. Strategic Organization, 13(4), 353–364.

    Article  Google Scholar 

  • Alwin, D. F. (1976). Assessing school effects: Some identities. Sociology of Education, 49(4), 294–303.

    Article  Google Scholar 

  • Alwin, D. F., & Hauser, R. M. (1975). The decomposition of effects in path analysis. American Sociological Review, 40, 37–47.

    Article  Google Scholar 

  • Barrick, M. R., Stewart, G. L., Neubert, M. J., & Mount, M. K. (1998). Relating member ability and personality to work-team processes and team effectiveness. Journal of Applied Psychology, 83(3), 377–391.

    Article  Google Scholar 

  • Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83(5), 762–765.

    Article  Google Scholar 

  • Bliese, P. D. (1998). Group size, ICC values, and group-level correlations: A simulation. Organizational Research Methods, 1(4), 355–373.

    Article  Google Scholar 

  • Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 349–381). San Francisco: Jossey-Bass.

    Google Scholar 

  • Bliese, P. D. (2006). Social climates: Drivers of soldier well-being and resilience. In A. B. Adller, C. A. Castro, & T. W. Britt (Eds.), Military life: The psychology of serving in peace and combat (Vol. 2. Operational stress, pp. 213–234). Westport: Praeger Security International.

    Google Scholar 

  • Bliese, P. D., & Halverson, R. R. (1996). Individual and nomothetic models of job stress: An examination of work hours, cohesion and well-being. Journal of Applied Social Psychology, 26(13), 1171–1189.

    Article  Google Scholar 

  • Bliese, P. D., & Hanges, P. J. (2004). Being both too liberal and too conservative: The perils of treating grouped data as though they were independent. Organizational Research Methods, 7(4), 400–417.

    Article  Google Scholar 

  • Bliese, P. D., Chan, D., & Ployhart, R. E. (2007). Multilevel methods: Future directions in measurement, longitudinal analyses, and nonnormal outcomes. Organizational Research Methods, 10(4), 551–563.

    Article  Google Scholar 

  • Bloom, H. S., Bos, J. M., & Lee, S. W. (1999). Using cluster random assignment to measure program impacts: Statistical implications for the evaluation of education programs. Evaluation Review, 23(4), 445–469.

    Article  PubMed  Google Scholar 

  • Chen, G., Mathieu, M. J., & Bliese, P. D. (2004). A framework for conducting multilevel construct validation. In F. Dansereau & F. J. Yammarino (Eds.), Research in multi-level issues, volume 3: Multi-level issues in organizational behavior and processes (pp. 273–303). Oxford: Elsevier Science.

    Chapter  Google Scholar 

  • Dansereau, F., Alutto, J. A., & Yammarino, F. J. (1984). Theory testing in organizational behavior: The varient approach. Englewood Cliffs Prentice-Hall.

    Google Scholar 

  • Firebaugh, G. (1978). A rule for inferring individual-level relationships from aggregate data. American Sociological Review, 43(4), 557–572.

    Article  Google Scholar 

  • Firebaugh, G. (1980). Groups as contexts and frog ponds. In K. H. Roberts & L. Burstein (Eds.), Issues in aggregation (pp. 43–52). San Francisco: Jossey-Bass.

    Google Scholar 

  • González-Morales, M. G., Peiró, J. M., Rodriguéz, I., & Bliese, P. D. (2012). Perceived collective burnout: A multilevel explanation of burnout. Anxiety, Stress, and Coping, 25(1), 43–61.

    Article  PubMed  Google Scholar 

  • Hedges, L., & Hedberg, E. C. (2007). Intraclass correlation values for planning group randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87.

    Article  Google Scholar 

  • Hofmann, D. A., & Gavin, M. B. (1998). Centering decisions in hierarchical linear models: Implications for research in organizations. Journal of Management, 24(5), 623–641.

    Article  Google Scholar 

  • Kreft, I., & De Leeuw, J. (1998). Introducing multilevel modeling. Thousand Oaks: Sage.

    Book  Google Scholar 

  • LaHuis, D. M., & Ferguson, M. W. (2009). The accuracy of significance tests for slope variance components in multilevel random coefficient models. Organizational Research Methods, 12(3), 418–435.

    Article  Google Scholar 

  • Lang, J., Thomas, J. L., Bliese, P. D., & Adler, A. B. (2007). Job demands and job performance: The mediating effect of psychological and physical strain and the moderating effect of role clarity. Journal of Occupational Health Psychology, 12, 116–124.

    Article  PubMed  Google Scholar 

  • Li, Y., Wang, Z., Yang, L.-Q., & Liu, S. (2016). The crossover of psychological distress from leaders to subordinates in teams: The role of abusive supervision, psychological capital, and team performance. Journal of Occupational Health Psychology, 21(2), 142–153.

    Article  PubMed  Google Scholar 

  • Liao, H., & Chuang, A. (2007). Transforming service employees and climate: A multilevel, multisource examination of transformational leadership in building long-term service relationships. Journal of Applied Psychology, 92(4), 1006–1019.

    Article  PubMed  Google Scholar 

  • Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., & Schabenberger, O. (2006). SAS for mixed models. Cary: SAS Institute.

    Google Scholar 

  • Liu, D., Liao, H., & Loi, R. (2012). The dark side of leadership: A three-level investigation of the cascading effect of abusive supervision on employee creativity. Academy of Management Journal, 55(5), 1187–1212.

    Article  Google Scholar 

  • LoPilato, A. C., & Vandenberg, R. J. (2015). The not so direct cross-level direct effect. In C. E. Lance & R. J. Vandenberg (Eds.), More statistical and methodological myths and urban legends (pp. 292–310). New York: Routledge.

    Google Scholar 

  • Luciano, M. M., Mathieu, J. E., & Ruddy, T. M. (2014). Leading multiple teams: Average and relative external leadership influences on team empowerment and effectiveness. Journal of Applied Psychology, 99(2), 322–331.

    Article  PubMed  Google Scholar 

  • Mathieu, J. E., & Kohler, S. S. (1990). A cross-level examination of group absence influences on individual absence. Journal of Applied Psychology, 75(2), 217–220.

    Article  Google Scholar 

  • McNeish, D., Stapleton, L. M., & Silverman, R. D. (2017). On the unnecessary ubiquity of hierarchical linear modeling. Psychological Methods, Advance online publication. doi:10.1037/met0000078.

    Google Scholar 

  • Miron-Spektor, E., Erez, M., & Naveh, E. (2011). The effect of conformist and attentive-to-detail members on team innovation: Reconciling the innovation paradox. Academy of Management Journal, 54(4), 740–760.

    Article  Google Scholar 

  • Morgeson, F. P., & Hofmann, D. A. (1999). The structure and function of collective constructs: Implications for multilevel resarch and theory development. Academy of Management Review, 24(2), 249–265.

    Google Scholar 

  • Morrison, E. W., Wheeler-Smith, S. L., & Kamdar, D. (2011). Speaking up in groups: A cross-level study of group voice climate and voice. Journal of Applied Psychology, 96(1), 183–191.

    Article  PubMed  Google Scholar 

  • Murray, D. M., & Short, B. (1995). Intra-class correlation among measures related to alcohol use by young adults: Estimates, correlates and applications in intervention studies. Journal of Studies on Alcohol, 56, 681–694.

    Article  PubMed  Google Scholar 

  • Ostroff, C. (1993). Comparing correlations based on individual-level and aggregated data. Journal of Applied Psychology, 78, 569–582.

    Article  Google Scholar 

  • Pinheiro, J. C., & Bates, D. M. (2000). Mixed-effects models in S and S-PLUS. New York: Springer-Verlag.

    Book  Google Scholar 

  • Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15(3), 209–233.

    Article  PubMed  Google Scholar 

  • Raudenbush, S. W. (2009). Adaptive centering with random effects: An alternative to the fixed effects model for studying time-varying treatments in school settings. Education, 4, 468–491.

    Google Scholar 

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks: Sage.

    Google Scholar 

  • Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357.

    Article  Google Scholar 

  • Sampson, R. J. (2003). The neighborhood context of well-being. Perspectives in Biology and Medicine, 46(3), S53–S64.

    Article  PubMed  Google Scholar 

  • Sampson, R. J., Raudenbush, S. W., & Earls, F. (1997). Neighborhoods and violent crime: A multilevel study of collective efficacy. Science, 277(5328), 918–924.

    Article  PubMed  Google Scholar 

  • Scherbaum, C. A., & Ferreter, J. M. (2009). Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organizational Research Methods, 12(2), 347–367.

    Article  Google Scholar 

  • Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks: Sage.

    Google Scholar 

  • Verbeek, M. (2008). A guide to modern econometrics. John Wiley & Sons.

  • Wang, X.-H., & Howell, J. M. (2010). Exploring the dual-level effects of transformational leadership on followers. Journal of Applied Psychology, 95(6), 1134–1144.

    Article  PubMed  Google Scholar 

  • Whitener, E. M. (2001). Do “high commitment” human resource practices affect employee commitment? A cross-level analysis using hierarchical linear modeling. Journal of Management, 27(5), 515–535.

    Google Scholar 

  • Wolfinger, R. D. (1997). An example of using mixed models and PROC MIXED for longitudinal data. Journal of Biopharmaceutical Statistics, 7(4), 481–500.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Darla Moore School of Business, University of South Carolina, Columbia, SC, USA

    Paul D. Bliese, Mark A. Maltarich & Jonathan L. Hendricks

Authors
  1. Paul D. Bliese
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark A. Maltarich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jonathan L. Hendricks
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paul D. Bliese.

Appendix: R Code

Appendix: R Code

ᅟ

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bliese, P.D., Maltarich, M.A. & Hendricks, J.L. Back to Basics with Mixed-Effects Models: Nine Take-Away Points. J Bus Psychol 33, 1–23 (2018). https://doi.org/10.1007/s10869-017-9491-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-017-9491-z

Keywords

Search

Navigation