Skip to main content

Advertisement

SpringerLink
Log in

Effect Sizes and Statistical Methods for Meta-Analysis in Higher Education

  • Published:
Research in Higher Education Aims and scope Submit manuscript

Abstract

Quantitative meta-analysis is a very useful, yet underutilized, technique for synthesizing research findings in higher education. Meta-analytic inquiry can be more challenging in higher education than in other fields of study as a result of (a) concerns about the use of regression coefficients as a metric for comparing the magnitude of effects across studies, and (b) the non-independence of observations that occurs when a single study contains multiple effect sizes. This methodological note discusses these two important issues and provides concrete suggestions for addressing them. First, meta-analysis scholars have concluded that standardized regression coefficients, which are often provided in higher education manuscripts, constitute an appropriate metric of effect size. Second, hierarchical linear modeling (HLM) analyses provide an effective method for conducting meta-analytic research while accounting for the non-independence of observations, and HLM is generally superior to other proposed methods that attempt to remedy this same problem. A discussion of how to implement these techniques appropriately is provided.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  • Afshartous, D. (1995). Determination of sample size for multilevel model design. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.

  • Alexander, R. A., Scozzaro, M. J., & Borodkin, L. J. (1989). Statistical and empirical examination of the Chi-square test for homogeneity of correlations in meta-analysis. Psychological Bulletin, 106, 329–331.

    Article  Google Scholar 

  • Becker, B. J., & Wu, M.-J. (2007). The synthesis of regression slopes in meta-analysis. Statistical Science, 22, 414–429.

    Article  Google Scholar 

  • Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. New York: Wiley.

    Book  Google Scholar 

  • Bornmann, L., Mutz, R., & Daniel, H.-D. (2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1, 226–238.

    Article  Google Scholar 

  • Bowman, N. A. (2010). College diversity experiences and cognitive development: A meta-analysis. Review of Educational Research, 80, 4–33.

    Article  Google Scholar 

  • Bowman, N. A. (2011). Promoting participation in a diverse democracy: A meta-analysis of college diversity experiences and civic engagement. Review of Educational Research, 81, 29–68.

    Article  Google Scholar 

  • Browne, W. J., & Draper, D. (2000). Implementation and performance issues in the Bayesian and likelihood fitting of multilevel models. Computational Statistics, 15, 391–420.

    Article  Google Scholar 

  • Busing, F. (1993). Distribution characteristics of variance estimates in two-level models (Technical Report PRM 93-04). Leiden, the Netherlands: Department of Psychometrics and Research Methodology, University of Leiden.

  • Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Cooper, H. M. (2009). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Cooper, H. M., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation.

    Google Scholar 

  • Crede, M., Roch, S. G., & Kieszczynka, U. M. (2010). Class attendance in college: A meta-analytic review of the relationship of class attendance with grades and student characteristics. Review of Educational Research, 80, 272–295.

    Article  Google Scholar 

  • Denson, N. (2009). Do curricular and co-curricular diversity activities influence racial bias? A meta-analysis. Review of Educational Research, 79, 805–838.

    Article  Google Scholar 

  • Denson, N., & Seltzer, M. H. (2011). Meta-analysis in higher education: An illustrative example using hierarchical linear modeling. Research in Higher Education, 52, 215–244.

    Article  Google Scholar 

  • Ethington, C. A. (1997). A hierarchical linear modeling approach to studying college effects. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 12, pp. 165–194). New York: Agathon Press.

    Google Scholar 

  • Gleser, L. J., & Olkin, I. (1994). Stochastically dependent effect sizes. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 339–356). New York: Russell Sage Foundation.

    Google Scholar 

  • Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.

    Google Scholar 

  • Hox, J. J., & de Leeuw, E. D. (2003). Multilevel models for meta-analysis. In S. P. Reise & N. Duan (Eds.), Multilevel modeling: Methodological advances, issues, and applications (pp. 90–111). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Kline, R. B. (2005). Principles and practice of structural equation modeling (2nd ed.). New York: Guilford Press.

    Google Scholar 

  • Kreft, I., & de Leeuw, J. (1998). Introducing multilevel modeling. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Kuncel, N. R., Crede, M., & Thomas, L. L. (2005). The validity of self-reported grade point averages, class ranks, and test scores: A meta-analysis and review of the literature. Review of Educational Research, 75, 63–82.

    Article  Google Scholar 

  • Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Newbury Park, CA: Sage Publications.

    Google Scholar 

  • Maas, C. J. M., & Hox, J. J. (2004). Robustness issues in multilevel regression analysis. Statistica Neerlandica, 58, 127–137.

    Article  Google Scholar 

  • Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1(3), 86–92.

    Article  Google Scholar 

  • Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., & O’Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multi-level approaches. Review of Educational Research, 79, 1290–1326.

    Article  Google Scholar 

  • Matt, G. E., & Cook, T. D. (1994). Threats to the validity of research syntheses. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 503–520). New York: Russell Sage Foundation.

    Google Scholar 

  • O’Mara, A. J., Marsh, H. W., Craven, R. G., & Debus, R. (2006). Do self-concept interventions make a difference? A synergistic blend of construct validation and meta-analysis. Educational Psychologist, 41, 181–206.

    Article  Google Scholar 

  • Peterson, R. A., & Brown, S. P. (2005). On the use of beta coefficients in meta-analysis. Journal of Applied Psychology, 90, 175–181.

    Article  Google Scholar 

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Newbury Park, CA: Sage Publications.

    Google Scholar 

  • Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 231–244). New York: Russell Sage Foundation.

    Google Scholar 

  • Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. In S. T. Fiske, D. Schacter, & C. Zahn-Waxler (Eds.), Annual review of psychology (Vol. 52, pp. 59–82). Palo Alto, CA: Annual Reviews.

    Google Scholar 

  • Snijders, T. A. B., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Stanley, T. D., & Jarrell, S. B. (2005). Meta-regression analysis: A quantitative method of literature surveys. Journal of Economic Surveys, 19, 299–308.

    Article  Google Scholar 

  • Van der Leeden, R., & Busing, F. (1994). First iteration versus IGLS/RIGLS estimates in two-level models: A Monte Carlo study with ML3 (Technical Report PRM 94-03). Department of Psychometrics and Research Methodology, University of Leiden, Leiden, the Netherlands.

  • Van der Leeden, R., Busing, F., & Meijer, E. (1997, April). Applications of bootstrap methods for two-level models. Paper presented at the Multilevel Conference, Amsterdam, Netherlands.

Download references

Acknowledgment

The author thanks Anat H. Levtov for her helpful comments on an earlier version of this manuscript.

Author information

Authors and Affiliations

  1. Department of Higher Education and Student Affairs, Bowling Green State University, 330 Education Building, Bowling Green, OH, 43403, USA

    Nicholas A. Bowman

Authors
  1. Nicholas A. Bowman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicholas A. Bowman.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bowman, N.A. Effect Sizes and Statistical Methods for Meta-Analysis in Higher Education. Res High Educ 53, 375–382 (2012). https://doi.org/10.1007/s11162-011-9232-5

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11162-011-9232-5

Keywords

Search

Navigation