Meta-Analysis: Conceptual Issues of Addressing Apparent Failure of Individual Study Replication or “Inexplicable” Heterogeneity

  • K. O’Rourke
Part of the Lecture Notes in Statistics book series (LNS, volume 148)


This paper is about issues of applying statistics to a particular area: meta-analysis (MA) of randomized clinical trials (RCTs) and possibly also observational clinical trials. Applying statistics is “messy”. As no theory or model is ever correct, when applying statistics we can only attempt to find the “least wrong” model or approach. I will identify some current approaches to MA as not being “least wrong”. This does not mean that they could not be “least wrong” for other applications. Discussing the applying of statistics— unfortunately for me at least—requires “wordy” explanations and I hope the reader will bear with me.


Prior Distribution Treatment Effect Estimate Treatment Effect Size Compound Distribution Flawed Study 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

12 References

  1. Bailey, K.R (1987). Inter-study differences: How should they influence the interpretations and analysis of results? Statist. Med. 6, 351–358.CrossRefGoogle Scholar
  2. Berkey, C.S., D.C Hoaglin, F. Mosteller, and G.A Colditz (1995). A random-effects regression model for meta-analysis. Statist. Med. 14, 395–411.CrossRefGoogle Scholar
  3. Casella, G. (1992). Conditional inference from confidence sets. In M. Ghosh and P. K. Pathak (Eds.), Current Issues in Statistical Inference: Essays in honor of D. Basu. Institute of Mathematical Statistics.Google Scholar
  4. Chatfield, C. (1995). Uncertainty, data mining and inference. J. Roy. Statist. Soc. Ser. A 158, 418–466.Google Scholar
  5. Cochran, W.G. (1937). Problems arising in the analysis of a series of similar experiments. J. Roy. Statist. Soc. Suppl. 4, 102–118.CrossRefGoogle Scholar
  6. Cox, D.R. (1958). The interpretation of the effects of nonadditivity in the Latin square. Biometrika 45, 69–73.zbMATHGoogle Scholar
  7. Cox, D.R. (1982). Combination of data. In S. Kotz and N. Johnson (Eds.), Encyclopedia Statisti. Sci. 2. New York: Wiley.Google Scholar
  8. Cox, D.R. and D.V. Hinkley (1974). Theoretical Statistics. London: Chapman and Hall.zbMATHGoogle Scholar
  9. Cox, D.R. and E.J. Snell (1981). Applied Statistics. London: Chapman and Hall.zbMATHCrossRefGoogle Scholar
  10. Cox, D.R. and E.J. Snell (1989). Analysis of Binary Data: Second edition. London: Chapman and Hall.Google Scholar
  11. Data Analysis Product Division, MathSoft (1997). S-Plus 4 guide to statistics. Seattle: Data Analysis Product Division, MathSoft.Google Scholar
  12. DerSimonian, R. and N. Laird (1986). Meta-analysis in clinical trials. Cont. Clin. Trials 7, 177–188.CrossRefGoogle Scholar
  13. Desmond, A.F. and V.P. Godambe (1999). Estimating functions. In P. Armitage and T. Colton (Eds.), Encyclopedia of Biostatistics. New York: Wiley.Google Scholar
  14. Detsky, A.S., C.D. Naylor, K. O’Rourke, A. McGeer, and K.A. L’Abbe (1992). Incorporating variations in the quality of individual randomized trials into meta-analysis. J. Clin. Epid. 45, 255–265.CrossRefGoogle Scholar
  15. DuMouchel, W., D. Pram, Z. Jin, S.L. Normand, B. Snow, S. Taylor, and R. Tweedie (1997). MetaGraphs: Software for Exploration and Modeling of Meta-analyses. Belmont Research Incorporated.Google Scholar
  16. Efron, B. (1996). Empirical Bayes methods for combining likelihoods (with discussion). J. Amer. Statist. Assoc. 91, 538–565.MathSciNetzbMATHCrossRefGoogle Scholar
  17. Emerson, J.D., D.C. Hoaglin, and F. Mosteller (1996). Simple robust procedures for combining risk differences in sets of 2×2 tables. Statist. Med. 15, 1465–1488.CrossRefGoogle Scholar
  18. Firth, D. (1990). Generalized linear models. In D. Hinkley, N. Reid, and E. Snell (Eds.), Statistical Theory and Modelling. London: Chapman and Hall.Google Scholar
  19. Fisher, R.A. (1937). The Design of Experiments. Edinburgh: Oliver and Boyd.zbMATHGoogle Scholar
  20. Fraser, D.A.S. (1976). Probability and Statistics: Theory and Application. North Scituate: Duxbury Press.Google Scholar
  21. Gardin, J.C. (1981). La Logic du Plausible: Essais d’Epistémologie Pratique. Ann Arbor, Michigan: University Microfilms International.Google Scholar
  22. Goodman, S.N. (1989). Meta-analysis and evidence. Cont. Clin. Trials 10, 188–204.CrossRefGoogle Scholar
  23. Greenland, S. and A. Salvan (1990). Bias in the one-step method for pooling study results. Statist. Med. 9, 247–252.CrossRefGoogle Scholar
  24. Grizzle, J.E. (1965). The two-period change over design and its use in clinical trials. Biometrics 21, 467–480.CrossRefGoogle Scholar
  25. Guttman, I., I. Olkin, and R. Philips (1993). Estimating the number of aberrant laboratories. Technical Report 9301, Department of Statistics, The University of Toronto.Google Scholar
  26. Jones, B. and J. Lewis (1995). The case for cross-over trials in phase III. Statist. Med. 14, 1025–1038.CrossRefGoogle Scholar
  27. L’Abbe, K.A., A.S. Detsky, and K. O’Rourke (1987). Meta-analysis in clinical research. Ann. Internal Med. 107, 224–233.Google Scholar
  28. Laird, N.M. and F. Mosteller (1990). Some statistical methods for combining experimental results. Internat. J. Techn. Assessment in Health Care 6, 5–30.CrossRefGoogle Scholar
  29. Lee, Y. and J.A. Neider (1996). Hierarchical generalized linear models. J. Roy. Statist. Soc. Ser. B 58, 619–678.MathSciNetzbMATHGoogle Scholar
  30. McCullagh, P. (2000). Re-sampling and exchangeable arrays. Bernouilli 6, 303–322.MathSciNetCrossRefGoogle Scholar
  31. Meier, P. (1987). Commentary. Statist. Med. 6, 329–331.CrossRefGoogle Scholar
  32. Morris, C.N. (1983). Parametric empirical Bayes inference: Theory and applications. J. Amer. Statist. Assoc. 78, 47–65.MathSciNetzbMATHCrossRefGoogle Scholar
  33. Morris, C.N. and S.L. Normand (1992). Hierarchical models for combining information and for meta-analysis. In J. Bernardo, J. Berger, A. Dawid, and A. Smith (Eds.), Bayesian Statistics 4. Oxford: Oxford University Press.Google Scholar
  34. Mosteller, F. and J.W. Tukey (1977). Data Analysis and Regression. New York: Addison-Wesley.Google Scholar
  35. Normand, S.L. (1995). Meta-analysis software: A comparative review. Amer. Statist. 49, 298–309.Google Scholar
  36. O’Rourke, K. and A.S. Detsky (1989). Meta-analysis in medical research: Strong encouragement for higher quality in individual research efforts. J. Clin. Epid. 42, 1021–1024.CrossRefGoogle Scholar
  37. O’Rourke, K., A. McGeer, C.D. Naylor, K.A. L’Abbe, and A.S. Detsky (1991). Incorporating quality appraisals into meta-analyses. Technical Report 9103, Department of Statistics, The University of Toronto.Google Scholar
  38. Pena, D. (1997). Combining information in statistical modeling. Amer. Statist. 51, 326–332.MathSciNetGoogle Scholar
  39. Peto, R. (1987). Discussion. Statist. Med. 6, 242.CrossRefGoogle Scholar
  40. Rao, C.R. (1988). Comment on Iyengar and Greenhouse’s “Selection models and the file drawer problem”. Statist. Sci. 3, 131.CrossRefGoogle Scholar
  41. Rubin, D.B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Edu. Psychology 66, 688–701.CrossRefGoogle Scholar
  42. Rubin, D.B. (1978). Bayesian inference for causal effects: The role of randomization. Annals of Statistics 6, 34–58.MathSciNetzbMATHCrossRefGoogle Scholar
  43. Rubin, D.B. (1990). Neyman (1923) and causal inference in experiments and observational studies. Statist. Sci. 5, 472–480.MathSciNetzbMATHGoogle Scholar
  44. Rubin, D.B. (1991). Practical implications of modes of statistical inference for causal effects and the critical role of the assignment mechanism. Biometrics 47, 1213–1234.MathSciNetzbMATHCrossRefGoogle Scholar
  45. Rubin, D.B. (1992). Meta-analysis: Literature synthesis or effect-size surface estimation? J. Edu. Statist. 17, 363–374.CrossRefGoogle Scholar
  46. Schulz, K.F. (1995). Unbiased research and the human spirit: The challenges of randomized controlled trials. Canad. Med. Assoc. J. 153, 783–786.Google Scholar
  47. Schulz, K.F., I. Chalmers, D.A. Grimes, and D.G. Altman (1994). Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. J. Amer. Med. Assoc. 272, 125–128.CrossRefGoogle Scholar
  48. Schulz, K.F., I. Chalmers, R.J. Hayes, and D.G. Altman (1995). Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. J. Amer. Med. Assoc. 273, 408–412.CrossRefGoogle Scholar
  49. Sen, S (1996). The AB/BA cross-over: How to perform the two stage analysis if you can’t be persuaded that you shouldn’t. In H. de Ridder (Ed.), Liber Amicorum Roel Van Strik, Erasmus University.Google Scholar
  50. Shapiro, S. (1997). Is meta-analysis a valid approach to the evaluation of small effects in observational studies? J. Clin. Epid. 50, 223–229.CrossRefGoogle Scholar
  51. Smith, T.C., D.J. Spiegelhalter, and M.H.K. Parmar (1996). Bayesian meta-analysis of randomized trials using graphical models and BUGS. In D. Berry and D. Stangl (Eds.), Bayesian Biostatistics. New York: Marcel Dekker.Google Scholar
  52. Tjur, T. (1998). Nonlinear regression, quasi likelihood, and over dispersion in generalized linear models. Amer. Statist. 52, 222–227.MathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2001

Authors and Affiliations

  • K. O’Rourke

There are no affiliations available

Personalised recommendations