Introduction
Scientists have known for centuries that a single study will not resolve a major issue. Indeed, a small sample study will not even resolve a minor issue. Thus, the foundation of a science is the culmination of knowledge from the results of many studies. (Hunter et al. 1982, p. 10)
Gene Glass’s meta-analytic work in psychotherapy and on class size in education marked the beginning of a new era in systematic reviews (Glass 1976; Glass and Smith 1979). Despite the initial objections of some skeptics (e.g., Eysenck 1978, 1995; Gallo 1978; Presby 1978), meta-analysis (M-A), as a methodology of quantitative synthesis, has continued to grow exponentially and has become the most reliable source of comprehensive summaries of quantitative research in a wide variety of fields (Glass 2016; Shadish and Lecy 2015).
However, like any other form of research, M-As can be biased in a number of ways (e.g., Abrami et al. 1988; Bernard et al. 2014a; Benard 2014; Cooper 2016; Polanin et al. 2016...
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abrami PC, Bernard RM (2012) Statistical control versus classification of study quality in meta-analysis. Eff Educ 4:43–72. https://doi.org/10.1080/19415532.2012.761889
Abrami PC, Cohen PA, d’Apollonia S (1988) Implementation problems in meta-analysis. Rev Educ Res 58:151–179. https://doi.org/10.3102/00346543058002151
Ahn S, Ames AJ, Myers ND (2012) A review of meta-analyses in education: methodological strengths and weaknesses. Rev Educ Res 82:436–476. https://doi.org/10.3102/0034654312458162
Benard RM (2014) Things I have learned about meta-analysis since 1990: reducing bias in search of “the big picture.”. Can J Learn Technol 40(3). https://doi.org/10.21432/T2MW29
Bernard RM, Naidu S (1990) Integrating research into instructional practice: the use and abuse of meta-analysis. Can J Educ Commun 19(3):171–195
Bernard RM, Abrami PC, Lou Y, Borokhovski E, Wade A, Wozney L, Huang B (2004) How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of educational research, 74(3):379–439.
Bernard RM, Borokhovski E, Schmid RF, Tamim RM (2014a) An exploration of bias in meta-analysis: the case of technology integration research in higher education. J Comput High Educ 26:183–209. https://doi.org/10.1007/s12528-014-9084-z
Bernard RM, Borokhovski E, Tamim R (2014b) Detecting bias in meta-analyses of distance education research: big pictures we can rely on. Distance Educ 35:271–293. https://doi.org/10.1080/01587919.2015.957433
Borenstein M, Hedges LV, Higgins JPT, Rothstein HR (2009) A basic introduction to fixed effect and random effects models for meta-analysis. Res Synth Methodol 1:97–111. https://doi.org/10.1002/jrsm.12
Card NA (2012) Applied meta-analysis for social sciences research. Guilford, New York
Cohen PA (1990) Things I have learned (so far). Am Psychol 54(12):1304–1312
Cook TD, Leviton LC (1980) Reviewing the literature: a comparison of traditional methods with meta-analysis. J Pers 48:449–472. https://doi.org/10.1111/j.1467-6494.1980.tb02379.x
Cooper H (2010) Research synthesis and meta-analysis (4th ed.). Thousand Oaks, CA: Sage Publications
Cooper H (2016) Research synthesis and meta-analysis: A step-by-step approach (Vol. 2). Sage publications
Cooper HM (2017) Research synthesis and meta-analysis: a step-by-step approach, 5th edn. Sage, Thousand Oaks
Cooper HM, Arkin R (1981) On quantitative reviewing. J Pers 49:225–230
Cooper H, Hedges LV, Valentine J (2009) The handbook of research synthesis and meta-analysis, 2nd edn. Russell Sage Foundation, New York
Duval S, Tweedie R (2000) A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. J Am Stat Assoc 95(449):89–98. https://doi.org/10.1080/01621459.2000.10473905
Eysenck HJ (1978) An exercise in mega-silliness. Am Psychol 33:517. https://doi.org/10.1037/0003-066X.33.5.517.a
Eysenck HJ (1995) Meta-analysis squared – does it make sense. Am Psychol 50:110–111. https://doi.org/10.1037/0003-066X.50.2.110
Gallo PS (1978) Meta-analysis – a mixed metaphor. Am Psychol 33:515–517
Glass GV (1976) Primary, secondary and meta-analysis of research. Educ Res 5(10):3–8. https://doi.org/10.3102/0013189X005010003
Glass GV (2016) One hundred years of research: prudent aspirations. Educ Res 45(2):69–72. https://doi.org/10.3102/0013189X16639026
Glass GV, Smith ML (1979) Meta-analysis of research on the relationship of class-size and achievement. Eval Policy Anal 1:2–16
Guzzo RA, Jackson SE, Katzell RA (1987) Meta-analysis analysis. Res Organ Behav 9:407–442
Hammerstrøm K, Wade A, Jørgensen AMK (2010) Searching for studies: a guide to information retrieval for Campbell systematic reviews, supplement 1. Retrieved from Campbell Collaboration website: http://www.campbellcollaboration.org/resources/research/new_information_retrieval_guide.php
Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic, Orlando
Higgins J, Lane PW, Anagnostelis B, Anzures-Cabrera J, Baker NF, Cappelleri JC, Whitehead A (2013) A tool to assess the quality of a meta-analysis. Res Synth Methods 4:351–366. https://doi.org/10.1002/jrsm.1092
Hunt M (1997) How science takes stock: the story of meta-analysis. Russell Sage Foundation, New York
Hunter JE, Schmidt FL, Jackson GB (1982) Meta-analysis. Beverly Hills, CA.
Hunter JE, Schmidt FL (2014) Methods of meta-analysis: correcting bias in research findings, 3rd edn. Sage, Newbury Park
Kline RB (2004) Beyond significance testing: reforming data analysis methods in behavioral research. American Psychological Association, Washington DC. https://doi.org/10.1037/10693-000
Lipsey MW (2003) Those confounded moderators in meta-analysis: Good, bad, and ugly. The Annals of the American Academy of Political and Social Science 587:69–81
Lipsey MW, Wilson D (2001) Practical meta-analysis. Sage Publications, Thousand Oaks
Meehl PE (1967) Theory-testing in psychology and physics: a methodological paradox. Philos Sci 34(2):103–115. https://doi.org/10.1086/288135
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF (1999) Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet 354:1896–1900
Pigott T (2012) Advances in meta-analysis. Springer, New York
Polanin JR, Tanner-Smith EE, Hennessy EA (2016) Estimating the difference between published and unpublished effect sizes: a meta-review. Rev Educ Res 86:207–236. https://doi.org/10.3102/0034654315582067
Presby S (1978) Overly broad categories obscure important differences between therapies. Am Psychol 33:514–515
Rothstein HR, Sutton AJ, Borenstein M (eds) (2005) Publication bias in meta-analysis – prevention, assessment and adjustments. Wiley, Chichester
Scammacca N, Roberts G, Stuebing KK (2014) Meta-analysis with complex research designs: dealing with dependence from multiple measures and multiple group comparisons. Rev Educ Res 84:328–364. https://doi.org/10.3102/0034654313500826
Schlosser RW, Wendt O, Sigafoos J (2007) Not all systematic reviews are created equal: considerations for appraisal. Evid Based Commun Assess Interv 1:138–150. https://doi.org/10.1080/17489530701560831
Seo H-J, Kim KU (2012) Quality assessment of systematic reviews or meta-analyses of nursing interventions conducted by Korean reviewers. BMC Med Res Methodol 12:129. https://doi.org/10.1186/1471-2288-12-129
Shadish WR, Lecy JD (2015) The meta-analytic big bang. Res Synth Methods 6:246–264. https://doi.org/10.1002/jrsm.1132
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Bouter LM (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 7:10. https://doi.org/10.1186/1471-2288-7-10
Slavin RE (1984) Meta-analysis in education: how has it been used? Educ Res 13:6–15
Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ et al (2010) Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 14(8). https://doi.org/10.3310/hta14080
Steiner DD, Lane IM, Dobbins GH, Schnur A, McConnell S (1991) A review of meta-analyses in organizational behavior and human resource management: an empirical assessment. Educ Psychol Meas 51:609–626. https://doi.org/10.1177/0013164491513008
Tamim, R. (2009). Effects of technology on students’ achievement: a second-order meta-analysis (doctoral dissertation). Available from ProQuest dissertations and theses database. (UMI No. NR63392)
Tamim R, Bernard RM, Borokhovski E, Abrami PC, Schmid RF (2011) What forty years of research says about the impact of technology on learning: a second-order meta-analysis and validation study. Rev Educ Res 81(1):4–28. https://doi.org/10.3102/0034654310393361
Tamim RM, Borokhovski E, Pickup DI, Bernard RM, El Saadi L (Under review). Tablets and smart mobile devices for teaching and learning: A systematic review and meta-analysis. Paper under review with Computers and Education
Tavakol M, Dennick R (2011) Making sense of Cronbach’s alpha. Int J Med Educ 2:53–55. https://doi.org/10.5116/ijme.4dfb.8dfd
Torgerson CJ (2007) The quality of systematic reviews of effectiveness in literacy learning in English: a ‘tertiary’ review. J Res Read 30:287–315
Valentine JC, Cooper HM (2008) A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: the study design and implementation assessment device (study DIAD). Psychol Methods 13:130–149. https://doi.org/10.1037/1082-989X.13.2.130
Valentine JC, Cooper H, Patall EA, Tyson D, Robinson JC (2010) A method for evaluating research syntheses: the quality, conclusions, and consensus of 12 syntheses of the effects of after-school programs. Res Synth Methods 1:20–38. https://doi.org/10.1002/jrsm.3
Yeşilyurt APDM (2010) Meta analysis of the computer assisted studies in science and mathematics: A sample of Turkey. The Turkish Online Journal of Educational Technology, 9(1):123–131
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Section Editor information
Appendix
Appendix
Study statistics and forest plot of 49 meta-analyses
Funnel plot of 49 meta-analyses
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this entry
Cite this entry
Tamim, R.M., Borokhovski, E.F., Bernard, R.M. (2020). Methodological Quality of Educational Technology Meta-analyses. In: Tatnall, A. (eds) Encyclopedia of Education and Information Technologies. Springer, Cham. https://doi.org/10.1007/978-3-030-10576-1_113
Download citation
DOI: https://doi.org/10.1007/978-3-030-10576-1_113
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-10575-4
Online ISBN: 978-3-030-10576-1
eBook Packages: Computer ScienceReference Module Computer Science and Engineering