Intensive Care Medicine

, Volume 39, Issue 8, pp 1386–1395 | Cite as

Quality of reporting of randomized controlled trials published in Intensive Care Medicine from 2001 to 2010

  • Nicola Latronico
  • Marta Metelli
  • Maddalena Turin
  • Simone Piva
  • Frank A. Rasulo
  • Cosetta Minelli
Original

Abstract

Purpose

To evaluate the methodological quality of randomized controlled trials (RCTs) published in Intensive Care Medicine from 2001 to 2010, and to compare it with a previous review of RCTs published from 1975 to 2000.

Methods

We assessed the quality of reporting of randomization, blinding and participant flow, both individually and combined within the Jadad scale, and compared them with findings from our previous review. For RCTs published from 2001 to 2010, we also evaluated the frequency of distorted finding presentation (spin) and inflated predicted treatment effect (delta inflation).

Results

In the 221 RCTs from 2001 to 2010, the sample size was significantly larger than in the older series, and there was a higher proportion of studies with negative findings. Reporting of the rationale for sample size estimation and allocation concealment increased significantly, but reporting of other important individual methodological components did not change substantially compared with the previous period and remained low. Among RCTs from 2001 to 2010, a spin strategy was used in 69 of 111 RCTs with statistically negative results, while delta inflation was present in 7 of 11 RCTs evaluating survival as a primary outcome. Papers with higher Jadad scores were cited more often than the others.

Conclusions

Quality of reporting of RCTs published in Intensive Care Medicine has only partly improved over time, and spin and delta bias are of frequent occurrence. There is a need for stronger adherence to CONSORT recommendations, with special emphasis on accurate description of randomization and blindness, and correct reporting of statistically non-significant results.

Keywords

Methodology Quality of reporting Randomization Impact factor Google scholar Scopus Web of Science 

Notes

Acknowledgments

We thank Prof. Massimo Antonelli, former Editor-in-Chief, and Prof. Jordi Mancebo, former Reviews Deputy Editor of Intensive Care Medicine, who suggested the investigation into the association between methodological quality of randomized controlled trials and frequency of citation.

Supplementary material

134_2013_2947_MOESM1_ESM.ppt (97 kb)
Supplementary material 1 (PPT 97 kb)

References

  1. 1.
    Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S (1995) Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials 16:62–73PubMedCrossRefGoogle Scholar
  2. 2.
    Schulz KF, Chalmers I, Hayes RJ, Altman DG (1995) Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273:408–412PubMedCrossRefGoogle Scholar
  3. 3.
    Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP (1998) Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 352:609–613PubMedCrossRefGoogle Scholar
  4. 4.
    Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T (2001) The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 134:663–694PubMedCrossRefGoogle Scholar
  5. 5.
    Kjaergard LL, Villumsen J, Gluud C (2001) Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 135:982–989PubMedCrossRefGoogle Scholar
  6. 6.
    Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, Ravaud P, Brorson S (2012) Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ 344:e1119PubMedCrossRefGoogle Scholar
  7. 7.
    Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, Als-Nielsen B, Balk EM, Gluud C, Gluud LL, JP AI, Schulz KF, Beynon R, Welton NJ, Wood L, Moher D, Deeks JJ, Sterne JA (2012) Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern MedGoogle Scholar
  8. 8.
    Jüni P, Altman DG, Egger M (2008) Assessing the quality of randomised controlled trialssystematic reviews in health care. BMJ, London, pp 87–108Google Scholar
  9. 9.
    Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ (1996) Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 17:1–12PubMedCrossRefGoogle Scholar
  10. 10.
    Altman DG (2001) Systematic reviews of evaluations of prognostic variables. BMJ 323:224–228PubMedCrossRefGoogle Scholar
  11. 11.
    Huwiler-Muntener K, Juni P, Junker C, Egger M (2002) Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 287:2801–2804PubMedCrossRefGoogle Scholar
  12. 12.
    Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336:601–605PubMedCrossRefGoogle Scholar
  13. 13.
    Aberegg SK, Richards DR, O’Brien JM (2010) Delta inflation: a bias in the design of randomized controlled trials in critical care medicine. Crit Care 14:R77PubMedCrossRefGoogle Scholar
  14. 14.
    Boutron I, Dutton S, Ravaud P, Altman DG (2010) Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA 303:2058–2064PubMedCrossRefGoogle Scholar
  15. 15.
    Piaggio G, Elbourne DR, Pocock SJ, Evans SJ, Altman DG (2012) Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement. JAMA 308:2594–2604PubMedCrossRefGoogle Scholar
  16. 16.
    Latronico N, Botteri M, Minelli C, Zanotti C, Bertolini G, Candiani A (2002) Quality of reporting of randomised controlled trials in the intensive care literature. A systematic analysis of papers published in Intensive Care Medicine over 26 years. Intensive Care Med 28:1316–1323PubMedCrossRefGoogle Scholar
  17. 17.
    Azoulay E, Citerio G, Timsit JF (2013) The identity of Intensive Care Medicine. Intensive Care Med 39:343–344CrossRefGoogle Scholar
  18. 18.
    Sud S, Sud M, Friedrich JO, Meade MO, Ferguson ND, Wunsch H, Adhikari NK (2010) High frequency oscillation in patients with acute lung injury and acute respiratory distress syndrome (ARDS): systematic review and meta-analysis. BMJ 340:c2327PubMedCrossRefGoogle Scholar
  19. 19.
    Felder TM, Palmer NR, Lal LS, Mullen PD (2011) What is the evidence for pharmaceutical patient assistance programs? A systematic review. J Health Care Poor Underserved 22:24–49PubMedGoogle Scholar
  20. 20.
    Poolman RW, Struijs PA, Krips R, Sierevelt IN, Lutz KH, Bhandari M (2006) Does a “Level I Evidence” rating imply high quality of reporting in orthopaedic randomised controlled trials? BMC Med Res Methodol 6:44PubMedCrossRefGoogle Scholar
  21. 21.
    Lai TY, Wong VW, Lam RF, Cheng AC, Lam DS, Leung GM (2007) Quality of reporting of key methodological items of randomized controlled trials in clinical ophthalmic journals. Ophthalmic Epidemiol 14:390–398PubMedCrossRefGoogle Scholar
  22. 22.
    Sut N, Senocak M, Uysal O, Koksalan H (2008) Assessing the quality of randomized controlled trials from two leading cancer journals using the CONSORT statement. Hematol Oncol Stem Cell Ther 1:38–43PubMedGoogle Scholar
  23. 23.
    Bai Y, Gao J, Zou DW, Li ZS (2009) Methodological reporting of randomized clinical trials in major gastroenterology and hepatology journals in 2006. Hepatology 49:2108–2112PubMedCrossRefGoogle Scholar
  24. 24.
    Danilla S, Wasiak J, Searle S, Arriagada C, Pedreros C, Cleland H, Spinks A (2009) Methodological quality of randomised controlled trials in burns care. A systematic review. Burns 35:956–961PubMedCrossRefGoogle Scholar
  25. 25.
    Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG (2010) The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ 340:c723PubMedCrossRefGoogle Scholar
  26. 26.
    Strech D, Soltmann B, Weikert B, Bauer M, Pfennig A (2011) Quality of reporting of randomized controlled trials of pharmacologic treatment of bipolar disorders: a systematic review. J Clin Psychiatr 72:1214–1221CrossRefGoogle Scholar
  27. 27.
    Agha RA, Camm CF, Edison E, Orgill DP (2012) The methodological quality of randomized controlled trials in plastic surgery needs improvement: a systematic review. J Plast Reconstr Aesthet SurgGoogle Scholar
  28. 28.
    Mills EJ, Wu P, Gagnier J, Devereaux PJ (2005) The quality of randomized trial reporting in leading medical journals since the revised CONSORT statement. Contemp Clin Trials 26:480–487PubMedCrossRefGoogle Scholar
  29. 29.
    Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, Dias S, Schulz KF, Plint AC, Moher D (2012) Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 11:MR000030Google Scholar
  30. 30.
    Schulz KF, Altman DG, Moher D (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med 152:726–732PubMedCrossRefGoogle Scholar
  31. 31.
    Dechartres A, Charles P, Hopewell S, Ravaud P, Altman DG (2011) Reviews assessing the quality or the reporting of randomized controlled trials are increasing over time but raised questions about how quality is assessed. J Clin Epidemiol 64:136–144PubMedCrossRefGoogle Scholar
  32. 32.
    Hirst A, Altman DG (2012) Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals. PLoS ONE 7:e35621PubMedCrossRefGoogle Scholar
  33. 33.
    Karanicolas PJ, Farrokhyar F, Bhandari M (2010) Practical tips for surgical research: blinding: who, what, when, why, how? Can J Surg 53:345–348PubMedGoogle Scholar
  34. 34.
    Schulz KF, Altman DG, Moher D, Fergusson D (2010) CONSORT 2010 changes and testing blindness in RCTs. Lancet 375:1144–1146PubMedCrossRefGoogle Scholar
  35. 35.
    Charles P, Giraudeau B, Dechartres A, Baron G, Ravaud P (2009) Reporting of sample size calculation in randomised controlled trials: review. BMJ 338:b1732PubMedCrossRefGoogle Scholar
  36. 36.
    Moher D, Fortin P, Jadad AR, Juni P, Klassen T, Le Lorier J, Liberati A, Linde K, Penna A (1996) Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet 347:363–366PubMedCrossRefGoogle Scholar
  37. 37.
    Schulz KF, Grimes DA (2002) Allocation concealment in randomised trials: defending against deciphering. Lancet 359:614–618PubMedCrossRefGoogle Scholar
  38. 38.
    Guyatt GH, Mills EJ, Elbourne D (2008) In the era of systematic reviews, does the size of an individual trial still matter. PLoS Med 5:e4PubMedCrossRefGoogle Scholar
  39. 39.
    Schulz KF, Grimes DA (2005) Multiplicity in randomised trials II: subgroup and interim analyses. Lancet 365:1657–1661PubMedCrossRefGoogle Scholar
  40. 40.
    Schulz KF, Grimes DA (2005) Multiplicity in randomised trials I: endpoints and treatments. Lancet 365:1591–1595PubMedCrossRefGoogle Scholar
  41. 41.
    Schulz KF, Grimes DA (2005) Sample size calculations in randomised trials: mandatory and mystical. Lancet 365:1348–1353PubMedCrossRefGoogle Scholar
  42. 42.
    Lundh A, Gotzsche PC (2008) Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies. BMC Med Res Methodol 8:22PubMedCrossRefGoogle Scholar
  43. 43.
    Hyman M (2010) Science for sale: protect yourself from medical research deception. http://www.huffingtonpost.com/dr-mark-hyman/dangerous-spin-doctors-7-_b_747325.html. Accessed 28 June 2012
  44. 44.
    International Committee of Medical Journal Editors (2013) Uniform requirements for manuscripts submitted to biomedical journals: publishing and editorial issues related to publication in biomedical journals: obligation to publish negative studies. http://www.icmje.org/publishing_1negative.html. Accessed 2 April 2013
  45. 45.
    Altman DG, Bland JM (1995) Absence of evidence is not evidence of absence. BMJ 311:485PubMedCrossRefGoogle Scholar
  46. 46.
    Powers JH (2008) Noninferiority and equivalence trials: deciphering ‘similarity’ of medical interventions. Stat Med 27:343–352PubMedCrossRefGoogle Scholar
  47. 47.
    Drazen JM (2012) Believe the data. N Engl J Med 367:1152–1153PubMedCrossRefGoogle Scholar
  48. 48.
    Kesselheim AS, Robertson CT, Myers JA, Rose SL, Gillet VBA, Ross KM, Glynn RJ, Joffe S, Avorn J (2012) A randomized study of how physicians interpret research funding disclosures. N Engl J Med 367:1119–1127Google Scholar
  49. 49.
    (2005) In praise of soft science. Nature 435:1003Google Scholar
  50. 50.
    Bonati MR, Drusini AG (1996) Morgagni and the impact factor. Nature 381:271–271PubMedCrossRefGoogle Scholar
  51. 51.
    Smith R (2003) Medical journals and pharmaceutical companies: uneasy bedfellows. BMJ 326:1202–1205PubMedCrossRefGoogle Scholar
  52. 52.
    Smith R (2005) Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Med 2:e138PubMedCrossRefGoogle Scholar
  53. 53.
    Lundh A, Barbateskovic M, Hrobjartsson A, Gotzsche PC (2010) Conflicts of interest at medical journals: the influence of industry-supported randomised trials on journal impact factors and revenue—cohort study. PLoS Med 7:e1000354PubMedCrossRefGoogle Scholar
  54. 54.
    McVeigh ME, Mann SJ (2009) The journal impact factor denominator: defining citable (counted) items. JAMA 302:1107–1109PubMedCrossRefGoogle Scholar
  55. 55.
    Callaham M, Wears RL, Weber E (2002) Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA 287:2847–2850PubMedCrossRefGoogle Scholar
  56. 56.
    Nieminen P, Carpenter J, Rucker G, Schumacher M (2006) The relationship between quality of research and citation frequency. BMC Med Res Methodol 6:42PubMedCrossRefGoogle Scholar
  57. 57.
    Etter JF, Stapleton J (2009) Citations to trials of nicotine replacement therapy were biased toward positive results and high-impact-factor journals. J Clin Epidemiol 62:831–837PubMedCrossRefGoogle Scholar
  58. 58.
    Filion KB, Pless IB (2008) Factors related to the frequency of citation of epidemiologic publications. Epidemiol Perspect Innov EP+I 5:3Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg and ESICM 2013

Authors and Affiliations

  • Nicola Latronico
    • 1
    • 2
  • Marta Metelli
    • 1
  • Maddalena Turin
    • 1
  • Simone Piva
    • 2
  • Frank A. Rasulo
    • 2
  • Cosetta Minelli
    • 3
  1. 1.University Division of Anesthesia and Critical Care Medicine, Department of Anesthesia, Critical Care and Perioperative MedicineUniversity of BresciaBresciaItaly
  2. 2.University Division of Neuroanesthesia and Neurocritical Care, Department of Anesthesia, Intensive Care and Perioperative MedicineUniversity of BresciaBresciaItaly
  3. 3.Respiratory Epidemiology and Public Health Group, National Heart and Lung InstituteImperial College LondonLondonUK

Personalised recommendations