Skip to main content
Log in

Methodology in meta–analysis: a study from Critical Care meta–analytic practice

  • Published:
Health Services and Outcomes Research Methodology Aims and scope Submit manuscript

Abstract

Methodological aspects of meta-analytic practice, heterogeneity, publication bias, metaregression and effect metric, were investigated in 14 meta-analyses reflecting major therapeutic concern in Critical Care practice.

Compared with the standard Q test, the exact Zelen test was more sensitive in identifying heterogeneity. Assessment of heterogeneity impact by the I 2 statistic was consistent with inferences afforded by both the Q and Zelen test. Publication bias was subject to test and metric determination: funnel plots exhibited variable asymmetry across studies and between metrics; the regression asymmetry test appeared more sensitive than the rank correlation test; the “trim and fill” method was the most sensitive, but suggested, on the basis of quantification of the effects of potentially missing studies, that meta-analyses may be resistant to such missingness. Metaregression of treatment effect against control risk using Bayesian hierarchical regression in all metrics (log odds ratio, log risk ratio and RD) suggested that naïve linear regression approaches over-diagnosed significant relationships and exhibited regression dilution.

Heterogeneity, publication bias and risk related treatment effects all demonstrate estimator and metric dependence; the RD metric would appear the most capricious in this regard.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Glass, G.V.: Primary, secondary, and meta-analysis of research. Educational Research 5, 3–8 (1976).

    Google Scholar 

  2. Egger, M., Smith, G.D., Altman, D.G.: Systematic reviews in health care: meta-analysis in context. 2nd ed. BMJ Publishing Group, London (2001).

    Google Scholar 

  3. Edwards, P., Clarke, M., DiGuiseppi, C., Pratap, S., Roberts, I., Wentz, R.: Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat Med 21, 1635–1640 (2002).

    Article  PubMed  Google Scholar 

  4. Greenland, S.: Invited commentary: A critical look at some popular meta-analytic methods. Am J Epidemiol 140, 290–296 (1994).

    CAS  PubMed  Google Scholar 

  5. Dickersin, K., Min, Y.I. Publication bias: The problem that won’t go away. Ann. N.Y. Acad. Sci. 703, 135–146 (1993).

    CAS  PubMed  Google Scholar 

  6. Terrin, N., Schmid, C.H., Lau, J., Olkin, I.: Adjusting for publication bias in the presence of heterogeneity. J. Clin. Epidemiol. 22, 2113–2126 (2003).

    Google Scholar 

  7. McIntosh, M.W.: The population risk as an explanatory variable in research synthesis of clinical trials. Stat. Med. 15, 1713–1728 (1996).

    Google Scholar 

  8. Schmid, C.H., Stark, P.C., Berlin, J.A., Landais, P., Lau, J.: Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors. J. Clin. Epidemiol. 57, 683–697 (2004).

    Article  PubMed  Google Scholar 

  9. Knaus, W.A., Harrell, F.E., Jr., LaBrecque, J.F., et al.: Use of predicted risk of mortality to evaluate the efficacy of anticytokine therapy in sepsis. The rhIL-1ra Phase III Sepsis Syndrome Study Group. Crit. Care. Med. 24, 46–56 (1996).

    Article  CAS  PubMed  Google Scholar 

  10. Minneci, P., Deans, K., Natanson, C., Eichacker, P.Q.: Increasing the efficacy of anti-inflammatory agents used in the treatment of sepsis. European Journal of Clinical Microbiology & Infectious Diseases 22, 1–9 (2003).

    CAS  Google Scholar 

  11. Knaus, W.A., Draper, E.A, Wagner, D.P, Zimmerman, J.E.: APACHE II: a severity of disease classification system. Crit. Care. Med. 13, 818–829 (1985).

    CAS  PubMed  Google Scholar 

  12. Eichacker, P.Q., Parent, C., Kalil, A., et al.: Risk and the efficacy of antiinflammatory agents: Retrospective and confirmatory studies of sepsis. Am. J. Respir. Crit. Care. Med. 166, 1197–1205 (2002).

    Article  PubMed  Google Scholar 

  13. Sun, X., Wagner, D.P., Knaus, W.A.: Does selective decontamination of the digestive tract reduce mortality for severely ill patients? Crit. Care. Med. 24, 753–755 (1996).

    Article  CAS  Google Scholar 

  14. Warn, D.E., Thompson, S.G., Spiegelhalter, D.J.: Bayesian random effects meta-analysis of trials with binary outcomes: methods for the absolute risk difference and relative risk scales. Stat. Med. 21, 1601–1623 (2002).

    Article  CAS  PubMed  Google Scholar 

  15. Gyldmark, M.: A review of cost studies of intensive care units: problems with the cost concept. Crit. Care. Med. 23, 964–972 (1995).

    Article  CAS  PubMed  Google Scholar 

  16. Steyerberg, E.W., Eijkemans, M.J., Harrell, F.E, Jr., Habbema, J.D.: Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat. Med. 19, 1059–1079 (2000).

    Article  CAS  PubMed  Google Scholar 

  17. Sterne, J.A., Bradburn, M.J., Egger, M.: Meta-analysis in Stata. In: Egger, M., Smith, G. D., and Altman, D. G. Systematic reviews in health care: meta-analysis in context. 2nd ed. BMJ Publishing Group, London (2001) pp. 347–385.

    Google Scholar 

  18. Cochran, W.G.: The combination of estimates from different experiments. Biometrics 10, 101–129 (1954).

    Google Scholar 

  19. DerSimonian, R., Laird, N.: Meta-analysis in clinical trials. Control Clin. Trials 7, 177–188 (1986).

    Article  CAS  PubMed  Google Scholar 

  20. Higgins, J.P., Thompson, S.G.: Quantifying heterogeneity in a meta-analysis. Stat. Med. 21, 1539–1558 (2002).

    Article  PubMed  Google Scholar 

  21. StatXact 4 for Windows: Statistical software for exact nonparametric inference. StatXact 4 for Windows: statistical software for exact nonparametric inference, 1999.

  22. Carpenter, J., Bithell, J.: Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat. Med. 19, 1141–1164 (2000).

    Article  CAS  PubMed  Google Scholar 

  23. Fox, J.: Describing univariate distributions. In: Fox, J. and Long, J.S. Modern methods of Data Analysis. Newbury Park, Sage Publications, CA (1990) pp. 58–125.

  24. Light, R., Pillemer, D.B.: Summing up: the science of reviewing research. Harvard University Press, Cambridge (1984).

    Google Scholar 

  25. Begg, C.B., Mazumdar, M.: Operating characteristics of a rank correlation test for publication bias. Biometrics 50, 1088–1101 (1994).

    CAS  PubMed  Google Scholar 

  26. Egger, M., Davey, S.G., Schneider, M., Minder, C.: Bias in meta-analysis detected by a simple, graphical test. BMJ 315, 629–634 (1997).

    CAS  PubMed  Google Scholar 

  27. Duval, S., Tweedie, R.: Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56, 455–463 (2000).

    Article  CAS  PubMed  Google Scholar 

  28. Steichen, T.: Tests for publication bias in meta-analysis. Stata Technical Bulletin Reprints 7, 125–133 (1998).

    Google Scholar 

  29. Steichen T.: Nonparametric “trim and fill” analysis of publication bias in meta-analysis. Stata Technical Bulletin 61, 8–14 (2000).

    Google Scholar 

  30. Sterne, J.A., Egger, M.: Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis. J. Clin. Epidemiol. 54, 1046–1055 (2001).

    Article  CAS  PubMed  Google Scholar 

  31. Sterne, J.A., Gavaghan, D., Egger, M.: Publication and related bias in meta-analysis: Power of statistical tests and prevalence in the literature. J. Clin. Epidemiol. 53, 1119–1129 (2000).

    Article  CAS  PubMed  Google Scholar 

  32. Sutton, A.J., Duval, S.J., Tweedie, R.L., Abrams, K.R., Jones, D.R.: Empirical assessment of effect of publication bias on meta-analyses. BMJ 320, 1574–1577 (2000).

    Article  CAS  PubMed  Google Scholar 

  33. Berlin, J.A., Antman, E.M.: Advantages and limitations of metaanalytic regressions of clinical trials data. Online J. Curr. Clin. Trials Doc. No. 134, 8425 (1994).

    Google Scholar 

  34. Sharp, S.J.: Meta-analysis regression. Stata Technical Bulletin Reprints 7, 148–155 (1998).

    Google Scholar 

  35. Speigelhalter, D., Thomas, A., Best, N.: WINBUGS. http://www.mrc-bsu.cam.ac.uk/bugs (2000).

  36. Gelman, A., Rubin, D.B.: Markov chain Monte Carlo methods in biostatistics. Stat. Methods Med. Res. 5, 339–355 (1996).

    CAS  PubMed  Google Scholar 

  37. Heyland, D.K., MacDonald, S., Keefe, L., Drover, J.W.: Total parenteral nutrition in the critically ill patient: A meta-analysis. JAMA 280, 2013–2019 (1998).

    Article  CAS  PubMed  Google Scholar 

  38. Cronin, L., Cook, D.J., Carlet, J., et al.: Corticosteroid treatment for sepsis: A critical appraisal and meta-analysis of the literature. Crit. Care. Med. 23, 1430–1439 (1995).

    Article  CAS  PubMed  Google Scholar 

  39. Selective Decontamination of the Digestive Tract Trailists’ Collaborative Group. Meta-analysis of randomised controlled trials of selective decontamination of the digestive tract. BMJ 307, 525–532 (1993).

    Google Scholar 

  40. Heyland, D.K., Cook, D.J., King, D., Kernerman, P., Brun-Buisson, C.: Maximizing oxygen delivery in critically ill patients: A methodologic appraisal of the evidence. Crit. Care. Med. 24, 517–524 (1996).

    Article  CAS  PubMed  Google Scholar 

  41. Keenan, S.P., Kernerman, P.D., Cook, D.J., Martin, C.M., McCormack, D., Sibbald, W.J.: Effect of noninvasive positive pressure ventilation on mortality in patients admitted with acute respiratory failure: a meta-analysis. Crit. Care. Med. 25, 1685–1692 (1997).

    Article  CAS  PubMed  Google Scholar 

  42. Cochrane Injuries Group Albumin Reviewers. Human albumin administration in citically ill patients: Systematic review of randomised controlled trials. BMJ 317, 235–240 (1998).

    Google Scholar 

  43. Cook, D.J., Reeve, B.K., Guyatt, G.H., et al.: Stress ulcer prophylaxis in critically ill patients. Resolving discordant meta-analyses. JAMA 275, 308–314 (1996).

    Article  CAS  PubMed  Google Scholar 

  44. Zeni, F., Freeman, B., Natanson, C.: Anti-inflammatory therapies to treat sepsis and septic shock: a reassessment. Crit. Care. Med. 25, 1095–1100 (1997).

    Article  CAS  PubMed  Google Scholar 

  45. Schierhout, G., Roberts, I.: Fluid resuscitation with colloid or crystalloid solutions in critically ill patients: A systematic review of randomised trials. BMJ 316, 961–964 (1998).

    CAS  PubMed  Google Scholar 

  46. Heys, S.D., Walker, L.G., Smith, I., Eremin, O.: Enteral nutritional supplementation with key nutrients in patients with critical illness and cancer: a meta-analysis of randomized controlled clinical trials. Ann. Surg. 229, 467–477 (1999).

    Article  CAS  PubMed  Google Scholar 

  47. Beale, R.J., Bryg, D.J., Bihari, D.J.: Immunonutrition in the critically ill: A systematic review of clinical outcome. Crit. Care. Med. 27, 2799–2805 (1999).

    Article  CAS  PubMed  Google Scholar 

  48. Zaloga, G.P.: Early enteral nutritional support improves outcome: Hypothesis or fact? Crit. Care. Med. 27, 259–261 (1999).

    Google Scholar 

  49. Wilkes, M.M., Navickis, R.J.: Patient survival after human albumin administration. A meta-analysis of randomized, controlled trials. Ann. Intern. Med. 135, 149–164 (2001).

    CAS  PubMed  Google Scholar 

  50. Heyland, D.K., Novak, F., Drover, J.W., Jain, M., Su, X., Suchner, U.: Should immunonutrition become routine in critically ill patients? A systematic review of the evidence. JAMA 286, 944–953 (2001).

    Article  CAS  PubMed  Google Scholar 

  51. Peter, J.V., Moran, J.L., Phillips-Hughes, J., Warn, D.: Noninvasive ventilation in acute respiratory failure—a meta-analysis update. Crit. Care. Med. 30, 555–562 (2002).

    Article  PubMed  Google Scholar 

  52. Sharp, S.J., Thompson, S.G.: Analysing the relationship between treatment effect and underlying risk in meta-analysis: comparison and development of approaches. Stat. Med. 19, 3251–3274 (2000).

    Article  CAS  PubMed  Google Scholar 

  53. Hahn, S., Garner, P., Williamson, P.: Are systematic reviews taking heterogeneity into account? An analysis from the Infectious Diseases Module of the Cochrane Library. J. Eval. Clin. Pract. 6, 231–233 (2000).

    Article  CAS  PubMed  Google Scholar 

  54. Thompson, S.G., Pocock, S.J.: Can meta-analyses be trusted? Lancet. 338, 1127–1130 (1991).

    Article  CAS  PubMed  Google Scholar 

  55. Fleiss, J.L.: Analysis of data from multiclinic trials. Control. Clin. Trials. 7, 267–275 (1986).

    Article  CAS  PubMed  Google Scholar 

  56. Emerson, J.D.: Combining estimates of the odds ratio: the state of the art. Stat. Methods Med. Res. 3, 157–178 (1994).

    CAS  PubMed  Google Scholar 

  57. Macaskill, P., Walter, S.D., Irwig, L.: A comparison of methods to detect publication bias in meta-analysis. Stat. Med. 20, 641–654 (2001).

    Article  CAS  PubMed  Google Scholar 

  58. Engels, E.A., Schmid, C.H., Terrin, N., Olkin, I., Lau, J.: Heterogeneity and statistical significance in meta-analysis: An empirical study of 125 meta-analyses. Stat. Med. 19, 1707–1728 (2000).

    Article  CAS  PubMed  Google Scholar 

  59. Fleiss, J.L.: The statistical basis of meta-analysis. Stat. Methods Med. Res. 2, 121–145 (1993).

    CAS  PubMed  Google Scholar 

  60. Lipsitz, S.R., Dear, K.B., Laird, N.M., Molenberghs, G.: Tests for homogeneity of the risk difference when data are sparse. Biometrics 54, 148–160 (1998).

    CAS  PubMed  Google Scholar 

  61. Lui, K.J., Kelly, C.: Tests for homogeneity of the risk ratio in a series of 2 × 2 tables. Stat. Med. 19, 2919–2932 (2000).

    Article  CAS  PubMed  Google Scholar 

  62. Poole, C., Greenland, S.: Random-effects meta-analyses are not always conservative. Am. J. Epidemiol. 150, 469–475 (1999).

    CAS  PubMed  Google Scholar 

  63. Bohning, D., Malzahn, U., Dietz, E., Schlattmann, P., Viwatwongkasem, C., Biggeri, A.: Some general points in estimating heterogeneity variance with the DerSimonian-Laird estimator. Biostatistics 3, 445–457 (2002).

    Article  PubMed  Google Scholar 

  64. Brockwell, S.E., Gordon, I.R.: A comparison of statistical methods for meta-analysis. Stat. Med. 20, 825–840 (2001).

    Article  CAS  PubMed  Google Scholar 

  65. Normand, S.L.: Meta-analysis: Formulating, evaluating, combining, and reporting. Stat. Med. 18, 321–359 (1999).

    Google Scholar 

  66. Takkouche, B., Cadarso-Suarez, C., Spiegelman, D.: Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis. Am. J. Epidemiol. 150, 206–215 (1999).

    CAS  PubMed  Google Scholar 

  67. Sterne, J.A., Juni, P., Schulz, K.F., Altman, D.G., Bartlett, C., Egger, M.: Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat. Med. 21, 1513–1524 (2002).

    Article  PubMed  Google Scholar 

  68. Hartung, J., Knapp, G.: A refined method for the meta-analysis of controlled clinical trials with binary outcome. Stat. Med. 20, 3875–3889 (2001).

    Article  CAS  PubMed  Google Scholar 

  69. Hoes, A.W., Grobbee, D.E., Lubsen, J.: Does drug treatment improve survival? Reconciling the trials in mild-to-moderate hypertension. J. Hypertens. 13, 805–811 (1995).

    CAS  PubMed  Google Scholar 

  70. Schmid, C.H., Lau, J., McIntosh, M.W., Cappelleri, J.C.: An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Stat. Med. 17, 1923–1942 (1998).

    Article  CAS  PubMed  Google Scholar 

  71. Sharp, S.J.: Analysing the relationship between treatment benefit and underlying risk: Precautions and recommendations. In: M. Egger, G.D. Smith, and D.G Altman, Systematic reviews in health care: Meta-analysis in context. 2nd ed. BMJ Publishing Group, London (2001) pp. 176–188.

  72. Arends, L.R., Hoes, A.W., Lubsden, J., Grobbee, D.E., Stijnen, T.: Baseline risk as predictor of treatment benefit: three clinical meta-re-analyses. Stat. Med. 19, 3497–3518 (2000).

    Article  CAS  PubMed  Google Scholar 

  73. Brand, R.: Importance of trends in the interpretation of an overall odds ratio in the meta-analysis of clinical trials. Stat. Med. 13, 295–296 (1994).

    Google Scholar 

  74. Sharp, S.J., Thompson, S.G., Altman, D.G.: The relation between treatment benefit and underlying risk in meta-analysis. BMJ 313, 735–738 (1996).

    CAS  PubMed  Google Scholar 

  75. Walter, S.D.: Variation in baseline risk as an explanation of heterogeneity in meta-analysis. Stat. Med. 16, 2883–2900 (1997).

    Article  CAS  PubMed  Google Scholar 

  76. Bernsen, R.M., Tasche, M.J., Nagelkerke, N.J.: Variation in baseline risk as an explanation of heterogeneity in meta-analysis by S. D. Walter, Statistics in Medicine, 16, 2883–2900 (1997). Stat. Med. 18, 233–238 (1999).

  77. Thompson, S.G., Sharp, S.J.: Explaining heterogeneity in meta-analysis: A comparison of methods. Stat. Med. 18, 2693–2708 (1999).

    Article  CAS  PubMed  Google Scholar 

  78. Schmid, C.H.: Exploring heterogeneity in randomized trials via meta-analysis. Drug. Inf. J. 33, 211–224 (1999).

    Google Scholar 

  79. van Houwelingen, H., Senn, S.: Investigating underlying risk as a source of heterogeneity in meta-analysis. Stat. Med. 18, 110–115 (1999).

    Article  CAS  PubMed  Google Scholar 

  80. Song, F., Eastwood, A.J., Gilbody, S., Duley, L., Sutton, A.J.: Publication and related biases. Health Technology Assessment (Rockville, Md.) 4, 1–115 (2000).

    Google Scholar 

  81. Terrin, N., Schmid, C.H., Lau, J.: In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J. Clin. Epidemiol. 58, 894–891 (2005).

    Article  PubMed  Google Scholar 

  82. Tang, J.L., Liu, J.L.: Misleading funnel plot for detection of bias in meta-analysis. J. Clin. Epidemiol. 53, 477–484 (2000).

    Article  CAS  PubMed  Google Scholar 

  83. Sterne, J.A., Gavaghan, D., Egger, M.: Publication and related bias in meta-analysis: Power of statistical tests and prevalence in the literature. J. Clin. Epidemiol. 53, 1119–1129 (2000).

    Article  CAS  PubMed  Google Scholar 

  84. Duval, S., Tweedie, R.: A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Society 95, 89–98 (2000).

    Google Scholar 

  85. Sterne, J.A., Egger, M.: High false positive rate for trim and fill method. BMJ http://bmj.com/cgi/eletters/320/7249/1574 (2000).

  86. Sutton, A.J.: Re: High false positive rate for trim and fill method. BMJ http://bmj.com/cgi/eletters/320/7249/1574 (2000).

  87. Deeks, J.J., Altman, D.G.: Effect measures for meta-analysis of trials with binary outcomes. In: M. Egger, G.D. Smith, and D.G. Altman, Systematic reviews in health care: Meta-analysis in context. 2nd ed. BMJ Publishing Group, London (2001) pp. 313–335.

    Google Scholar 

  88. Walter, S.D.: Choice of effect measure for epidemiological data. J. Clin. Epidemiol. 53, 931–939 (2000).

    Article  CAS  PubMed  Google Scholar 

  89. Zhang, J., Yu, K.: What’s the relative risk?: A mehtod of correcting the odds ratio in cohort studies of common outcomes. JAMA 290, 1690–1691 (1998).

    Article  Google Scholar 

  90. Lesaffre, E., Pledger, G.: A note on the number needed to treat. Control Clin. Trials 20, 439–447 (1999).

    Article  CAS  PubMed  Google Scholar 

  91. Hutton, J.L.: Number need to treat: properties and problems. Journal of the Royal Statistical Society, Series. A. 163, 403–419 (2000).

    Google Scholar 

  92. Smeeth, L., Haines, A., Ebrahim, S.: Numbers needed to treat derived from meta-analyses–sometimes informative, usually misleading. BMJ 318, 1548–1551 (1999).

    CAS  PubMed  Google Scholar 

  93. Deeks, J.J., Altman, D.G., Dooley, G., Sackett, D.L.: Choosing an appropriate dichotomous effect measure fro meta-analysis: Empirical evidence of the appropriateness of the odds ratio and relative risk. Control Clin. Trials 18, 84S–85S (1997).

    Article  Google Scholar 

  94. Deeks, J.J.: Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes. Stat. Med. 21, 1575–1600 (2002).

    Article  PubMed  Google Scholar 

  95. Smith, G.D., Song, F., Sheldon, T.A.: Cholesterol lowering and mortality: The importance of considering initial level of risk. BMJ 306, 1367–1373 (1993).

    Article  CAS  PubMed  Google Scholar 

  96. Cook, D., Guyatt, G.: Colloid use for fluid resuscitation: Evidence and spin. Ann. Intern. Med. 135, 205–208 (2001).

    CAS  PubMed  Google Scholar 

  97. Liberati, A., D’Amico, R., Pifferi, S., et al.: Antibiotics for preventing respiratory tract infections in adults receiving intensive care. [update of Cochrane Database Syst Rev. (2), CD000022 (2000)]. Cochrane Database of Systematic Reviews CD000022 (2000).

  98. Choi, P.T., Yip, G., Quinonez, L.G., Cook, D.J.: Crystalloids vs. colloids in fluid resuscitation: A systematic review. Crit. Care. Med. 27, 200–210 (1999).

    Article  CAS  PubMed  Google Scholar 

  99. The SAFE Study Investigators. A Comparison of Albumin and Saline for Fluid Resuscitation in the Intensive Care Unit. N. Engl. J. Med. 350, 2247–2256 (2004).

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to John L. Moran MBBS, FRACP.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Moran, J.L., Solomon, P.J. & Warn, D.E. Methodology in meta–analysis: a study from Critical Care meta–analytic practice. Health Serv Outcomes Res Method 5, 207–226 (2004). https://doi.org/10.1007/s10742-006-6829-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10742-006-6829-9

Keywords

Navigation