Skip to main content

Advertisement

SpringerLink
Log in

HLM Behind the Curtain: Unveiling Decisions Behind the Use and Interpretation of HLM in Higher Education Research

  • Research and Practice
  • Published:
Research in Higher Education Aims and scope Submit manuscript

Abstract

Hierarchical linear modeling (HLM) has become increasingly popular in the higher education literature, but there is significant variability in the current approaches to the conducting and reporting of HLM. The field currently lacks a general consensus around important issues such as the number of levels of analysis that are important to include and how much variance should be accounted for at each level in order for the HLM analysis to have practical significance (Dedrick et al., Rev Educ Res 79:69–102, 2009). The purpose of this research is to explore the use of a 3-level HLM model, appropriate contextualizing of results of HLM, and the interpretation of HLM results that resonates with practice. We used an example of a 3-level model from the National Study of Living Learning Programs to highlight the practical issues that arise in the interpretation of HLM within a higher education context.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. While the OLS regression and HLM analyses included slightly different samples (due to the handling of missing data and exclusion of 1-unit groups in HLM), these are the actual samples that would have been used had we chosen to conduct the regression or the HLM analyses. As the purpose is to compare practical scenarios, we decided to compare these two analyses despite the different sample sizes.

References

  • Astin, A. W. (1991). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education. New York: American Council on Education/Macmillan. doi:10.1007/BF00137472.

    Google Scholar 

  • Astin, A. W., & Denson, N. (2009). Multi-campus studies of college impact: Which statistical method is appropriate? Research in Higher Education, 50, 354–367. doi:10.1007/s1162-009-9121-3.

    Article  Google Scholar 

  • Campbell, C. M., Smith, M., Dugan, J., & Komives, S. (2012). Mentors and college student leadership outcomes: The importance of position and process. Review of Higher Education, 35(4), 595–625.

    Article  Google Scholar 

  • Cheslock, J., & Rios-Aguilar, C. (2008). Reaping (or not) the benefits of hierarchical data. Paper presented at the annual meeting of the Association for the Study of Higher Education, Jacksonville, FL.

  • Cheslock, J., & Rios-Aguilar, C. (2011). Multilevel analysis in higher education research: A multidisciplinary approach. In J.C. Smart & M.B. Paulson (Eds.), Higher education: handbook of theory and research (vol 26, pp. 85–123). Dordrecht: Kluwer Academic Publishers. doi: 10.1007/978-94-007-0702-3_3.

  • Clark, B. R., & Trow, M. (1966). The organizational context. In T. M. Newcomb & E. K. Wilson (Eds.), College peer groups: Problems and prospects for research. Chicago: Aldine.

    Google Scholar 

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: Lawrence Erlbaum Associates.

    Google Scholar 

  • Cox, B. E., McIntosh, K. L., Terenzini, P. T., Reason, R. D., & Lutovsky Quaye, B. R. (2010). Pedagogical signals of faculty approachability: Factors shaping faculty–student interaction outside the classroom. Research in Higher Education, 51(8), 767–788. doi:10.1007/s11162-010-9178-z.

    Article  Google Scholar 

  • Dedrick, R., et al. (2009). Multilevel modeling: A review of methodological issues and applications. Review of Educational Research, 79(1), 69–102. doi:10.3102/0034654308325581.

    Article  Google Scholar 

  • Di Bartolo, A., Dor, A., Fagioli, L. P., Garcia, H., Graves, D. M., Truong, K.K., & Thomas, S.L. (2011). Quantitative Methods in Higher Education Research: An Assessment of the Field at the Turn of the 21 st Century. Paper presented at the annual meeting of the Association for the Study of Higher Education, Charlotte, NC.

  • Dresel, M., & Rinderman, H. (2011). Counseling university instructors based on student evaluations of their teaching effectiveness: A multilevel test of its effectiveness under consideration of bias and unfairness variables. Research in Higher Education, 52(7), 717–737. doi:10.1007/s11162-011-9214-7.

    Article  Google Scholar 

  • Hausmann, L., Schofield, J., & Woods, R. (2007). Sense of belonging as a predictor of intentions to persist among African American and White first-year college students. Research in Higher Education, 48(7), 803–839. doi:10.1007/s11162-007-9052-9.

    Article  Google Scholar 

  • Hedges, L. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370. doi:10.3102/1076998606298043.

    Article  Google Scholar 

  • Hu, S., & Kuh, G. D. (2003). Maximizing what students get out of college: Testing a learning productivity model. Journal of College Student Development, 44(2), 185–203. doi:10.1353/csd.2003.0016.

    Article  Google Scholar 

  • Inkelas, K. K. (2007). National Study of living–learning programs 2007 Report Findings. http://drum.lib.umd.edu/bitstream/1903/10044/1/2007%20NSLLP%20Final%20Report.pdf.

  • Ishitani, T. T. (2011). The determinants of out-migration among in-state college students in the United States. Research in Higher Education, 52, 107–122. doi:10.1007/s11162-010-9187-y.

    Article  Google Scholar 

  • Johnson, D. R., Soldner, M., Brown Leonard, J., Alvarez, P., Inkelas, K. K., Rowan-Kenyon, H., et al. (2007). Examining sense of belonging among first-year undergraduates from different racial/ethnic groups. Journal of College Student Development, 48(5), 525–542. doi:10.1353/csd.20076.0054.

    Article  Google Scholar 

  • Kim, M. M. (2001). Institutional effectiveness of women-only colleges: Cultivating students’ desire to influence social conditions. The Journal of Higher Education, 72(3), 287–321.

    Article  Google Scholar 

  • Kim, M. M., & Conrad, C. F. (2006). The impact of Historically Black Colleges and Universities on the academic success of African–American students. Research in Higher Education, 47(4), 399–427. doi:10.1007/s11162-005-9001-4.

    Article  Google Scholar 

  • Kinzie, J., Thomas, A. D., Palmer, M. M., Umbach, P. D., & Kuh, G. D. (2007). Women students at coeducational and women’s colleges: How do their experiences compare? Journal of College Student Development, 48(2), 145–165. doi:10.1353/csd.2007.0015.

    Article  Google Scholar 

  • Kugelmass, H., & Ready, D. D. (2011). Racial/ethnic disparities in collegiate cognitive gains: A multilevel analysis of institutional influences on learning and its equitable distribution. Research in Higher Education, 52(4), 323–348. doi:10.1007/s11162-010-9200-5.

    Article  Google Scholar 

  • Kuh, G. (2008). High-impact educational practices: What they are, who has access to them, and why they matter. Washington DC: Association of American Colleges and Universities.

    Google Scholar 

  • Luke, D. A. (2004). Multilevel modeling. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1(3), 86–92. doi:10.1027/1614-1881.1.3.86.

    Google Scholar 

  • McCoach, D. B., & Black, A. C. (2008). Evaluation of model fit and adequacy. In A. A. O’Connell & D. B. McCoach (Eds.), Multilevel modeling of educational data (pp. 245–272). Charlotte, NC: Information Age Publishing, Inc.

    Google Scholar 

  • Moerbeek, M. (2004). The consequences of ignoring a level of nesting in multilevel analysis. Multivariate Behavioral Research, 39(1), 129–149.

    Article  Google Scholar 

  • Moos, R. H. (1979). Architectural, organizational, and contextual influences on living groups. Evaluating educational environments (pp. 67–90). San Francisco: Jossey-Bass.

    Google Scholar 

  • Myers, C. B. (2011). Union status and faculty job satisfaction: Contemporary evidence from the 2004 National Study of Postsecondary Faculty. The Review of Higher Education, 34(4), 657–684. doi:10.1353/rhe.2011.0028.

    Article  Google Scholar 

  • Nora, A., & Cabrera, A. F. (1993). The construct validity of institutional commitment: A confirmatory factor analysis. Research in Higher Education, 34, 243–262.

    Article  Google Scholar 

  • O’Connell, A. A., & McCoach, D. B. (Eds.). (2008). Multilevel modeling of educational data. Charlotte, NC: Information Age Publishing, Inc.

    Google Scholar 

  • Park, J. J., & Denson, N. (2009). Attitudes and advocacy: Understanding faculty views on racial/ethnic diversity. The Journal of Higher Education, 80(4), 415–438. doi:10.1353/jhe.0.0054.

    Article  Google Scholar 

  • Pascarella, E. T., et al. (2006). Institutional selectivity and good practices in undergraduate education: How strong is the link? The Journal of Higher Education, 77(2), 251–285. doi:10.1353/jhe.2006.0016.

    Article  Google Scholar 

  • Pike, G. R., Kuh, G. D., McCormick, A. C., Ethington, C. A., & Smart, J. C. (2011). If and when money matters: The relationships among educational expenditures, student engagement and students’ learning outcomes. Research in Higher Education, 52, 81–106. doi:10.1007/s11162-010-9183-2.

    Article  Google Scholar 

  • Porter, S. R. (2006). Institutional structures and student engagement. Research in Higher Education, 47(5), 521–558. doi:10.1007/s11162-005-9006-z.

    Article  Google Scholar 

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Roberts, J. K. & Monaco, J. P. (2006, April). Effect size measures for the two-level linear multilevel model. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.

  • Smeby, J. C., & Try, S. (2005). Departmental contexts and faculty research activity in Norway. Research in Higher Education, 46, 593–619. doi:10.1007/s11162-004-4136-2.

    Article  Google Scholar 

  • Snijders, T. A. B., & Bosker, R. J. (1994). Modeled variance in two-level models. Sociological Methods & Research, 22(3), 342–363.

    Article  Google Scholar 

  • Steinberg, M. P., Piraino, P., & Haveman, R. (2009). Access to higher education: Exploring the variation in Pell Grant prevalence among U.S. colleges and universities. The Review of Higher Education, 32(2), 235–270. doi:10.1353/rhe.0.0058.

    Article  Google Scholar 

  • Toutkoushian, R. K. (2001). Changes in demand for Public Land-Grant Universities: The use of panel datasets. Research in Higher Education, 42, 679–707. doi:10.1023/A:1012201511122.

    Article  Google Scholar 

  • Umbach, P. D., Palmer, M. M., Kuh, G. D., & Hannah, S. J. (2006). Intercollegiate athletes and effective educational practices: Winning combination or losing effort? Research in Higher Education, 47(6), 709–733. doi:10.1007/s11162-006-9012-9.

    Article  Google Scholar 

  • Volkwein, J. F., Cabrera, A. F., Lee, C., & Strauss, L. C. (2005). Classroom Instructional Vitality: Two Competing Explanations. Paper presented at the annual meeting of the Association for the Study of Higher Education, Philadelphia, PA.

  • Walsh, W. B. (1989). Person-environment interaction. In G. D. Kuh, J. P. Bean, D. Hossler, & F. K. Stage (Eds.), ASHE reader on college students (pp. 105–113). Needham Heights: Ginn Press.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Nebraska, Lincoln, NE, USA

    Elizabeth Niehaus

  2. Teachers College, Columbia University, New York, NY, USA

    Corbin M. Campbell

  3. University of Virginia, Charlottesville, VA, USA

    Karen Kurotsuchi Inkelas

Authors
  1. Elizabeth Niehaus
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Corbin M. Campbell
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karen Kurotsuchi Inkelas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elizabeth Niehaus.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Niehaus, E., Campbell, C.M. & Inkelas, K.K. HLM Behind the Curtain: Unveiling Decisions Behind the Use and Interpretation of HLM in Higher Education Research. Res High Educ 55, 101–122 (2014). https://doi.org/10.1007/s11162-013-9306-7

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11162-013-9306-7

Keywords

Search

Navigation