Skip to main content
SpringerLink
Log in

Analyzing multilevel data: comparing findings from hierarchical linear modeling and ordinary least squares regression

  • Published:
Higher Education Aims and scope Submit manuscript

Abstract

This study examined the differing conclusions one may come to depending upon the type of analysis chosen, hierarchical linear modeling or ordinary least squares (OLS) regression. To illustrate this point, this study examined the influences of seniors’ self-reported critical thinking abilities three ways: (1) an OLS regression with the student as the unit of analysis, (2) an OLS regression with the institution as the unit of analysis, and (3) a three-level hierarchical linear model. Overall, results demonstrate that one would come to substantively different conclusions regarding the influences on students’ perceived critical thinking ability depending upon the type of analysis chosen, especially in regards to the effects of the institutional characteristics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Anaya, G. (1996). College experiences and student learning: The influence of active learning, college environments, and cocurricular activities. Journal of College Student Development, 37, 611–622.

    Google Scholar 

  • Astin, A. W. (1993). What matters in college? Four critical years revisited. San Francisco: Jossey-Bass.

    Google Scholar 

  • Bauer, K. W., & Liang, Q. (2003). The effect of personality and precollege characteristics on first-year activities and academic performance. Journal of College Student Development, 44(3), 277–290.

    Article  Google Scholar 

  • Becher, T. (1994). The significance of disciplinary differences. Studies in Higher Education, 19(2), 151–161.

    Article  Google Scholar 

  • Biglan, A. (1973a). The characteristics of subject matter in different academic areas. Journal of Applied Psychology, 57(3), 195–203.

    Article  Google Scholar 

  • Biglan, A. (1973b). Relationships between subject matter characteristics and the structure and output of university departments. Journal of Applied Psychology, 57(3), 204–213.

    Article  Google Scholar 

  • Bowman, N. A. (2011). Examining systematic errors in predictors of college student self-reported gains. In S. Herzog & Bowman, N. A. (Eds.), New Directions for Institutional Research, 150, 7–19.

  • Braxton, J. (1995). Disciplines with an affinity for the improvement of undergraduate education. In N. Hativa & M. Marincovich (Eds.), Disciplinary differences in teaching and learning. San Francisco: Jossey-Bass Publishers.

    Google Scholar 

  • Brint, S., Cantwell, A. M., & Saxena, P. (2010). Disciplinary categories, majors, and undergraduate academic experiences: Rethinking Bok’s “Underachieving Colleges” thesis. Research in Higher Education, 53(1), 1–25.

    Article  Google Scholar 

  • Burstein, L. (1980). The analysis of multilevel data in educational research and evaluation. In D. C. Berliner (Ed.), Review of research in education (Vol. 8). Washington, DC: American Educational Research Association.

    Google Scholar 

  • Carini, R. M., Kuh, G. D., & Klein, S. P. (2006). Student engagement and student learning: Testing the linkages. Research in Higher Education, 47(1), 1–32.

    Article  Google Scholar 

  • Cheslock, J. J., & Rios-Aguilar, C. (2011). Multilevel analysis in higher education research: A multidisciplinary approach. In J. C. Smart & M. B. Paulsen (Eds.), Higher education: Handbook of theory and research (Vol. 26, pp. 85–123). New York: Agathon Press.

    Chapter  Google Scholar 

  • Doyle, S., Edison, M., & Pascarella, E. (1998, November). The “seven principles of good practice in undergraduate education” as process indicators of cognitive development in college: A longitudinal study. Paper presented at the meeting of the Association for the Study of Higher Education, Miami.

  • Draper, D. (1995). Inference and hierarchical modeling in the social sciences. Journal of Educational and Behavioral Statistics, 20(2), 115–147.

    Google Scholar 

  • Ethington, C. A. (1997). A hierarchical linear modeling approach to studying college effects. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 12, pp. 165–194). New York: Agathon Press.

    Google Scholar 

  • Facione, N. (1997). Critical thinking assessment in nursing education programs: An aggregate data analysis. Millbrae, CA: California Academic Press.

    Google Scholar 

  • Facione, P. (2000). The California critical thinking skills test (CCTST). Millbrae, CA: California Academic Press.

    Google Scholar 

  • Gellin, A. (2003). The effect of undergraduate student involvement on critical thinking: A meta-analysis of the literature 1991–2000. Journal of College Student Development, 44(6), 746–762.

    Article  Google Scholar 

  • Gonyea, R. M., & Miller, A. (2011). Clearing the AIR about the use of self-reported gains in institutional research. In S. Herzog & Bowman, N. A. (Eds.), New Directions for Institutional Research, 150, 99–111.

  • Hagedorn, L. S., Pascarella, E., Edison, M., Braxton, J., Nora, A., & Terenzini, P. (1999). Institutional context and the development of critical thinking: A research note. The Review of Higher Education, 22(3), 265–285.

    Article  Google Scholar 

  • Hannan, M., & Burstein, L. (1974). Estimation from grouped observations. American Sociological Review, 39, 374–392.

    Article  Google Scholar 

  • Hofmann, D. A. (1997). An overview of the logic and rationale of hierarchical linear models. Journal of Management, 23(6), 723–744.

    Article  Google Scholar 

  • Hofmann, D. A.,& Gavin, M. B. (1998). Centering decisions in hierarchical linear models: Implications for research in organizations. Journal of Management, 24(5), 623–641.

    Google Scholar 

  • Hurtado, S., & Ponjuan, L. (2005). Latino educational outcomes and the campus climate. Journal of Hispanic Higher Education, 4(3), 235–251.

    Article  Google Scholar 

  • Kim, Y. K., & Sax, L. J. (2009). Student-faculty interaction in research universities: Differences by student gender, race, social class and first-generation status. Research in Higher Education, 50(5), 437–459.

    Article  Google Scholar 

  • Kim, Y. K., & Sax, L. J. (2011). Are the effects of student–faculty interaction dependent on academic major? An examination using multilevel modeling. Research in Higher Education, 52(6), 589–615.

    Article  Google Scholar 

  • King, P. M., Wood, P. K., & Mines, R. A. (1990). Critical thinking among college and graduate students. The Review of Higher Education, 13(2), 167–186.

    Google Scholar 

  • Kreft, I., & de Leeuw, J. (1998). Introducing multilevel modeling. Newbury Park, CA: Sage.

    Google Scholar 

  • Kuh, G. D. (2001). The National Survey of Student Engagement: Conceptual framework and overview of psychometric properties. Bloomington, IN: Indiana University Center for Postsecondary Research.

    Google Scholar 

  • Kuh, G. D., & Hu, S. (1999). Is more better? Student-faculty interaction revisited. Paper presented at the meeting of the Association for the Study of Higher Education, San Antonio, TX.

  • Li, G., Long, S., & Simpson, M. E. (1999). Self-perceived gains in critical thinking and communication skills: Are there disciplinary differences? Research in Higher Education, 40(1), 43–60.

    Article  Google Scholar 

  • Locks, A., Hurtado, S., Bowman, N., & Oseguera, L. (2008). Extending notions of campus climate and diversity to the transition to college. Review of Higher Education, 31(3), 257–285.

    Article  Google Scholar 

  • McCormick, A. C., Pike, G. R., Kuh, G. D., & Chen, P. D. (2009). Comparing the utility of the 2000 and 2005 Carnegie classification systems in research on students’ college experiences and outcomes. Research in Higher Education, 50(2), 144–167.

    Article  Google Scholar 

  • National Survey of Student Engagement. (2006). NSSE 2006 overview. Retrieved December 21, 2012 from http://nsse.iub.edu/pdf/2006_Institutional_Report/NSSE%202006%20Overview.pdf.

  • Osborne, J. W. (2000). Advantages of hierarchical linear modeling. Practical Assessment, Research & Evaluation, 7(1), 1–33.

    Google Scholar 

  • Pascarella, E. (1989). The development of critical thinking: Does college make a difference? Journal of College Student Development, 30(1), 19–26.

    Google Scholar 

  • Pascarella, E., Cruse, T., Umbach, P., Wolniak, G., Kuh, G., Carini, R., et al. (2006). Institutional selectivity and good practices in undergraduate education: How strong is the link? The Journal of Higher Education, 77(2), 251–285.

    Article  Google Scholar 

  • Pascarella, E., & Terenzini, P. (1991). How college affects students: Findings and insights from twenty years of research. San Francisco: Jossey-Bass.

    Google Scholar 

  • Pascarella, E., & Terenzini, P. (2005). How college affects students: A third decade of research. San Francisco: Jossey-Bass.

    Google Scholar 

  • Pedhazur, E. J. (1997). Multiple regression in behavioral research (3rd ed.). New York: Holt, Rinehart and Winston.

    Google Scholar 

  • Pike, G., & Killian, T. (2001). Reported gains in student learning: Do academic disciplines make a difference? Research in Higher Education, 42(4), 429–454.

    Article  Google Scholar 

  • Pike, G., Kuh, G., & McCormick, A. (2011). An investigation of the contingent relationships between learning community participation and student engagement. Research in Higher Education, 52(3), 300–322.

    Article  Google Scholar 

  • Porter, S. R. (2012). Self-reported learning gains: A theory and test of college student survey response. Research in Higher Education. doi:10.1007/s11162-012-9277-0.

  • Quinn, C., Harding, H., Matkin, G., & Burbach, M. E. (2010, May). Students’ self-perceived critical thinking skills in an agricultural ethics course. Proceedings of the 2010 American Association for agricultural education research conference, Omaha, NE. Retrieved November 28, 2012 from http://hfi.unl.edu/cq/quinn%202010%20students%20self-perceived%20critical%20thinking%20skills%20in%20an%20ag%20ethics%20course.pdf.

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2010). HLM 7 for windows [Computer software]. Lincolnwood, IL: Scientific Software International, Inc.

    Google Scholar 

  • Rudd, R., Baker, M., & Hoover, T. (2000). Undergraduate agriculture student learning styles and critical thinking abilities: Is there a relationship? Journal of Agriculture Education, 41(3), 2–12.

    Google Scholar 

  • Smart, J. C. (2005). Perspectives of the editor: Attributes of exemplary research manuscripts employing quantitative analyses. Research in Higher Education, 46(4), 461–477.

    Article  Google Scholar 

  • Smart, J. C., Feldman, K. A., & Ethington, C. A. (2000). Academic disciplines: Holland’s theory and the study of college students and faculty. Nashville, TN: Vanderbilt University Press.

    Google Scholar 

  • Toutkoushian, R. K., & Smart, J. C. (2001). Do institutional characteristics affect student’s gains from college? The Review of Higher Education, 25(1), 39–61.

    Article  Google Scholar 

  • Tsui, L. (1999). Courses and instruction affecting critical thinking. Research in Higher Education, 40(2), 185–200.

    Google Scholar 

  • Tsui, L. (2002). Fostering critical thinking through effective pedagogy. The Journal of Higher Education, 73(6), 740–763.

    Article  Google Scholar 

  • Watson, G., & Glaser, E. M. (2006). Watson-Glaser critical thinking appraisal, short form manual. San Antonio, TX: Pearson.

    Google Scholar 

  • Whitt, E., Edison, M., Pascarella, E., Nora, A., & Terenzini, P. (1999). Interaction with peers and objective and self-reported cognitive outcomes across three years of college. Journal of College Student Development, 35, 198–207.

    Google Scholar 

  • Willms, J. D., & Raudenbush, S. W. (1989). A longitudinal hierarchal linear model for estimating school effects and their stability. Journal of Educational Measurement, 26(3), 209–232.

    Article  Google Scholar 

  • Winter, D., McClelland, D., & Stewart, A. (1981). A new case for the liberal arts: Assessing institutional goals and student development. San Francisco: Jossey-Bass.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Education, Indiana University Center for Postsecondary Research, 1900 E. Tenth Street, Suite 419, Bloomington, IN, 47406, USA

    Louis M. Rocconi

Authors
  1. Louis M. Rocconi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Louis M. Rocconi.

Appendix

Appendix

See Table 7.

Table 7 List of majors and Biglan (1973a, b) classification
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rocconi, L.M. Analyzing multilevel data: comparing findings from hierarchical linear modeling and ordinary least squares regression. High Educ 66, 439–461 (2013). https://doi.org/10.1007/s10734-013-9615-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10734-013-9615-y

Keywords

Search

Navigation