Abstract
In the context of multiple treatments for a particular problem or disorder, it is important theoretically and clinically to investigate whether any one treatment is more effective than another. Typically researchers report the results of the comparison of two treatments, and the meta-analytic problem is to synthesize the various comparisons of two treatments to test the omnibus null hypothesis that the true differences of all particular pairs of treatments are zero versus the alternative that there is at least one true nonzero difference. Two tests, one proposed by Wampold et al. (Psychol. Bull. 122:203–215, 1997) based on the homogeneity of effects, and one proposed here based on the distribution of the absolute value of the effects, were investigated. Based on a Monte Carlo simulation, both tests adequately maintained nominal error rates, and both demonstrated adequate power, although the Wampold test was slightly more powerful for non-uniform alternatives. The error rates and power were essentially unchanged in the presence of random effects. The tests were illustrated with a reanalysis of two published meta-analyses (psychotherapy and antidepressants). It is concluded that both tests are viable for testing the omnibus null hypothesis of no treatment differences.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Benish S., Imel Z.E., Wampold B.E.: The relative efficacy of bona fide psychotherapies of post-traumatic stress disorder: a meta-analysis of direct comparisons. Clin. Psychol. Rev. 28, 746–758 (2008)
Borenstein, M.: Effect sizes for continuous data. In: Cooper, H., Hedges, L.V. & Valentine J.C. (eds.) The handbook of research synthesis and meta-analysis, 2nd ed., pp. 221–235. Russel Sage Foundation, New York (2009)
Box G.E.P., Muller M.E.: A note on the generation of random normal deviates. Ann. Math. Stat. 28, 610–611 (1958)
Cipriani A., Furukawa T.A., Salanti G., Geddes J.R., Higgins J.P.T., Churchill R. et al.: Comparative efficacy and acceptability of 12 new-generation antidepressants: a multiple-treatments meta-analysis. Lancet 373, 746–758 (2009)
Cooper, H., Hedges, L.V. (eds.): The handbook of research synthesis. Russell Sage Foundationm, New York (1994)
Cooper, H., Hedges, L.V., Valentine, J.C.: The handbook of research synthesis and meta-analysis (2nd ed.). Russel Sage Foundation, New York (2009)
Cramér, H.: Mathematical methods of statistics. Princeton University Press, Princeton (1946)
Crits-Christoph, P.: Limitations of the dodo bird verdict and the role of clinical trials in psychotherapy research: comment on Wampold et al. Psychol. Bull. 122, 216–220 (1997)
Cuijpers P., van Straten A., Andersson G., Oppen P.: Psychotherapy for depression in adults: a meta-analysis of comparative outcome studies. J. Consult. Clin. Psychol. 76, 909–922 (2008)
Geary R.C.: The ratio of the mean deviation to the standard deviation as a test of normality. Biometrika 27, 310–332 (1935)
Glass G.V.: Primary, secondary, and meta-analysis of research. Educ. Res. 5, 3–8 (1976)
Gloaguen V., Cottraux J., Cucherat M., Blackburn I.: A meta-analysis of the effects of cognitive therapy in depressed patients. J. Affect. Disord. 49, 59–72 (1998)
Hedges L.V.: Distribution theory for Glass’s estimator of effect size and related estimators. J. Educ. Stat. 6(2), 107–128 (1981). doi:10.2307/1164588
Hedges, L.V., Olkin I.: Statistical methods for meta-analysis. Academic Press, San Diego (1985)
Howard K.I., Krause M.S., Saunders S.M., Kopta S.M.: Trials and tribulations in the meta-analysis of treatment differences: comment on Wampold et al. (1997). Psychol. Bull. 122, 221–225 (1997)
Hunt, M.: How science takes stock: the story of meta-analysis. Russell Sage Foundation, New York (1997)
Imel, Z.E., Wampold, B.E., Miller, S.D., Fleming, R.R.: Distinctions without a difference: direct comparisons of sychotherapies for alcohol use disorders. J. Addict. Behav. 533–543 (2008)
Mann C.C.: Can meta-analysis make policy?. Science 266, 960–962 (1994)
Miller S.D., Wampold B.E., Varhely K.: Direct comparisons of treatment modalities for youth disorders: a meta-analysis. Psychother. Res. 18, 5–14 (2008)
Pearson K.: On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to hove arisen from random sampling. Philos. Mag. 50, 157–175 (1900)
Press, W.H., Flannery, B.P., Teukolsky, S.A., Vetterling, W.T.: Numerical recipes in FORTRAN: the art of scientific computing, vol. 1. Cambridge Univeresity Press, Cambridge (1992)
Raudenbush, S.W.: Analyzing effect sizes: random-effects models. In: Cooper, H., Hedges, L.V. & Valentine J.C. (eds.) The handbook of research synthesis and meta-analysis, 2nd ed., pp. 295–316. Russel Sage Foundation, New York (2009)
Robinson L.A., Berman R.A., Neimeyer J.S.: Psychotherapy for the treatment of depression: a comprehensive review of controlled outcome research. Psychol. Bull. 108, 30–49 (1990)
Shapiro D.A., Shapiro D.: Meta-analysis of comparative therapy outcome research: a critical appraisal. Behav. Psychother. 10, 4–25 (1982)
Shapiro D.A., Shapiro D.: Meta-analysis of comparative therapy outcome studies: a replication and refinement. Psychol. Bull. 92, 581–604 (1982)
Siev J., Huppert J., Chambless D.L.: The dodo bird, treatment technique, and disseminating empirically supported treatments. Behav. Ther. 32, 69–75 (2009)
Wampold, B.E.: The great psychotherapy debate: model, methods, and findings. Lawrence Erlbaum Associates, Mahwah (2001)
Wampold B.E., Imel Z.E., Laska K.M., Benish S., Miller S.D., Flûckiger C. et al.: Determining what works in the treatment of PTSD. Clin. Psychol. Rev. 30, 923–933 (2010)
Wampold B.E., Imel Z.E., Miller S.D.: Barriers to the dissemination of empirically supported treatments: matching messages to the evidence. Behav. Ther 32, 144–155 (2009)
Wampold B.E., Minami T., Baskin T.W., Tierney S.C.: A meta-(re)analysis of the effects of cognitive therapy versus “other therapies” for depression. J. Affect. Disord. 68, 159–165 (2002)
Wampold B.E., Mondin G.W., Moody M., Stich F., Benson K., Ahn H.: A meta-analysis of outcome studies comparing bona fide psychotherapies: empirically, “All must have prizes”. Psychol. Bull. 122, 203–215 (1997)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wampold, B.E., Serlin, R.C. Meta-analytic methods to test relative efficacy. Qual Quant 48, 755–765 (2014). https://doi.org/10.1007/s11135-012-9800-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-012-9800-6