Usability and sensitivity of the risk of bias assessment tool for randomized controlled trials of pharmacist interventions

  • Fernanda S. Tonin
  • Livia A. Lopes
  • Inajara Rotta
  • Aline F. Bonetti
  • Roberto Pontarolo
  • Cassyano J. Correr
  • Fernando Fernandez-LlimosEmail author
Research Article


Background The Cochrane collaboration risk of bias assessment (RoB) tool is used in several fields to evaluate the methodological quality of studies. Its strengths and challenges are discussed. Objective To assess the sensitivity of the RoB tool in studies of pharmacist interventions. Setting DEPICT database was used to pool randomized controlled trials (RCTs) of complex interventions. Method A Guide for RoB Judgment in Pharmacy Services was created to help in the interpretation and judgment of bias criteria. The evaluation of bias (low, unclear, high risk) was performed by RCT. Sensitivity analyses were performed to assess the influence of different interpretations of eight elements of judgment in the RoB tool. Paired analysis and estimations of the effect size (95% confidence interval) of the criteria modifications compared to the original analyses were calculated. Main outcome measure Changes in the interpretations of judgment in the RoB tool. Results Overall, 8.3, 45.4, and 46.3% of the studies were determined to have low, unclear, and high risk of bias, respectively. High risk of bias was caused by attrition and detection domains. The number of studies classified with high risk of bias significantly increased for five of the eight interpretations, while unclear risk of bias increased for three interpretations (with a negligible effect size in all of them). Lack of blinding, loss of participants, and the use of subjective and self-reported outcomes were the main elements resulting in high risk of bias. Conclusion The RoB tool is useful for evaluating RCTs of pharmacist interventions if adapted criteria for judgment are used. Ignoring these adjustments produces a floor-effect with studies classified with high risk of bias.


Cochrane collaboration Methodology Outcome assessment Pharmacists Risk of bias RoB tool 



This work was supported by Brazilian National Council of Technological and Scientific Development (CNPq), Coordination for the Improvement of Higher Education Personnel (CAPES). The funding sources had no role in the study design, data collection, data analyses, data interpretation, or writing of the report. The corresponding author had full access to all of the data in the study and was responsible for making the final decision to submit the manuscript for publication.

Conflicts of interest

All authors declare that they have no conflict of interest.

Supplementary material

11096_2019_818_MOESM1_ESM.docx (38 kb)
Supplementary material 1 (DOCX 37 kb)


  1. 1.
    Armijo-Olivo S, Fuentes J, Ospina M, Saltaji H, Hartling L. Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: a descriptive analysis. BMC Med Res Methodol. 2013;13:116.CrossRefGoogle Scholar
  2. 2.
    Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet. 2017;390:415–23.CrossRefGoogle Scholar
  3. 3.
    Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs Seida J, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012.CrossRefGoogle Scholar
  4. 4.
    Viswanathan M, Ansari MT, Berkman ND, Chang S, Hartling L, McPheeters M, et al. Assessing the risk of bias of individual studies in systematic reviews of health care interventions. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ: Rockville, 2008.Google Scholar
  5. 5.
    Juni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ. 2001;323:42–6.CrossRefGoogle Scholar
  6. 6.
    Sylvester RJ, Canfield SE, Lam TB, Marconi L, MacLennan S, Yuan Y, et al. Conflict of evidence: resolving discrepancies when findings from randomized controlled trials and meta-analyses disagree. Eur Urol. 2017;71:811–9.CrossRefGoogle Scholar
  7. 7.
    Zeng X, Zhang Y, Kwong JS, Zhang C, Li S, Sun F, et al. The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: a systematic review. J Evid Based Med. 2015;8:2–10.CrossRefGoogle Scholar
  8. 8.
    Jordan VM, Lensen SF, Farquhar CM. There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review. J Clin Epidemiol. 2017;81:72–6.CrossRefGoogle Scholar
  9. 9.
    Mansournia MA, Higgins JP, Sterne JA, Hernan MA. Biases in randomized trials: a conversation between trialists and epidemiologists. Epidemiology. 2017;28:54–9.CrossRefGoogle Scholar
  10. 10.
    Yamato TP, Maher C, Koes B, Moseley A. The PEDro scale had acceptably high convergent validity, construct validity and inter-rater reliability in evaluating methodological quality of pharmaceutical trials. J Clin Epidemiol. 2017;86:176–81.CrossRefGoogle Scholar
  11. 11.
    Hempel S, Miles JN, Booth MJ, Wang Z, Morton SC, Shekelle PG. Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis. Syst Rev. 2013;2:107.CrossRefGoogle Scholar
  12. 12.
    Saltaji H, Ospina MB, Armijo-Olivo S, Agarwal S, Cummings GG, Amin M, et al. Evaluation of risk of bias assessment of trials in systematic reviews of oral health interventions, 1991–2014: a methodology study. J Am Dent Assoc. 2016;147:720–8.CrossRefGoogle Scholar
  13. 13.
    Vale CL, Tierney JF, Burdett S. Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews. BMJ. 2013;346:f1798.CrossRefGoogle Scholar
  14. 14.
    Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG. Assessment of study quality for systematic reviews: a comparison of the Cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: methodological research. J Eval Clin Pract. 2012;18:12–8.CrossRefGoogle Scholar
  15. 15.
    Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther. 2008;88:156–75.CrossRefGoogle Scholar
  16. 16.
    Higgins JPT, Green S. In: Cochrane Handbook for systematic reviews of interventions version 5.1.0. Accessed 09 Jan 2019.
  17. 17.
    da Costa BR, Beckett B, Diaz A, Resta NM, Johnston BC, Egger M, et al. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study. Syst Rev. 2017;6:44.CrossRefGoogle Scholar
  18. 18.
    Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013;66:973–81.CrossRefGoogle Scholar
  19. 19.
    Manchikanti L, Hirsch JA, Cohen SP, Heavner JE, Falco FJ, Diwan S, et al. Assessment of methodologic quality of randomized trials of interventional techniques: development of an interventional pain management specific instrument. Pain Physician. 2014;17:E263–90.Google Scholar
  20. 20.
    Jorgensen L, Paludan-Muller AS, Laursen DR, Savovic J, Boutron I, Sterne JA, et al. Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews. Syst Rev. 2016;5:80.CrossRefGoogle Scholar
  21. 21.
    Higgins J, Sterne J, Savović J, Page M, Hróbjartsson A, Boutron I, et al. A revised tool for assessing risk of bias in randomized trials. Cochrane Database Syst Rev. 2016;10:30–1.Google Scholar
  22. 22.
    Vo TH, Charpiat B, Catoire C, Juste M, Roubille R, Rose FX, et al. Tools for assessing potential significance of pharmacist interventions: a systematic review. Drug Saf. 2016;39:131–46.CrossRefGoogle Scholar
  23. 23.
    Tsuyuki RT. Designing pharmacy practice research trials. Can J Hosp Pharm. 2014;67:226–9.Google Scholar
  24. 24.
    Correr CJ, Melchiors AC, de Souza TT, Rotta I, Salgado TM, Fernandez-Llimos F. A tool to characterize the components of pharmacist interventions in clinical pharmacy services: the DEPICT project. Ann Pharmacother. 2013;47:946–52.CrossRefGoogle Scholar
  25. 25.
    Rotta I, Salgado TM, Felix DC, Souza TT, Correr CJ, Fernandez-Llimos F. Ensuring consistent reporting of clinical pharmacy services to enhance reproducibility in practice: an improved version of DEPICT. J Eval Clin Pract. 2015;21:584–90.CrossRefGoogle Scholar
  26. 26.
    Rotta I, Souza TT, Salgado TM, Correr CJ, Fernandez-Llimos F. Characterization of published randomized controlled trials assessing clinical pharmacy services around the world. Res Soc Adm Pharm. 2017;13:201–8.CrossRefGoogle Scholar
  27. 27.
    da Costa BR, Resta NM, Beckett B, Israel-Stahre N, Diaz A, Johnston BC, et al. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol. Syst Rev. 2014;13(3):144.CrossRefGoogle Scholar
  28. 28.
    Moustgaard H, Bello S, Miller FG, Hrobjartsson A. Subjective and objective outcomes in randomized clinical trials: definitions differed in methods publications and were often absent from trial reports. J Clin Epidemiol. 2014;67:1327–34.CrossRefGoogle Scholar
  29. 29.
    Hoffman JI. The incorrect use of Chi square analysis for paired data. Clin Exp Immunol. 1976;24:227–9.Google Scholar
  30. 30.
    Cliff N. Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull. 1993;114:494–509.CrossRefGoogle Scholar
  31. 31.
    Dreier M, Borutta B, Stahmeyer J, Krauth C, Walter U. Comparison of tools for assessing the methodological quality of primary and secondary studies in health technology assessment reports in Germany. GMS Health Technol Assess. 2010;6:Doc07.Google Scholar
  32. 32.
    Armijo-Olivo S, da Costa BR, Cummings GG, Ha C, Fuentes J, Saltaji H, et al. PEDro or Cochrane to assess the quality of clinical trials? A meta-epidemiological study. PLoS ONE. 2015;10:e0132634.CrossRefGoogle Scholar
  33. 33.
    Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282:1054–60.CrossRefGoogle Scholar
  34. 34.
    Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ. 2004;328:22–4.CrossRefGoogle Scholar
  35. 35.
    Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;2010(340):c332.CrossRefGoogle Scholar
  36. 36.
    Dechartres A, Trinquart L, Faber T, Ravaud P. Empirical evaluation of which trial characteristics are associated with treatment effect estimates. J Clin Epidemiol. 2016;77:24–37.CrossRefGoogle Scholar
  37. 37.
    Glynn LG, Murphy AW, Smith SM, Schroeder K, Fahey T. Self-monitoring and other non-pharmacological interventions to improve the management of hypertension in primary care: a systematic review. Br J Gen Pract. 2010;60:e476–88.CrossRefGoogle Scholar
  38. 38.
    Bos JM, van den Bemt PM, de Smet PA, Kramers C. The effect of prescriber education on medication-related patient harm in the hospital: a systematic review. Br J Clin Pharmacol. 2017;83:953–61.CrossRefGoogle Scholar
  39. 39.
    Clark L, Fairhurst C, Torgerson DJ. Allocation concealment in randomised controlled trials: Are we getting better? BMJ. 2016;355:i5663.CrossRefGoogle Scholar
  40. 40.
    Clark T, Davies H, Mansmann U. Five questions that need answering when considering the design of clinical trials. Trials. 2014;15:286.CrossRefGoogle Scholar
  41. 41.
    Groenwold RH, Moons KG, Vandenbroucke JP. Randomized trials with missing outcome data: how to analyze and what to report. CMAJ. 2014;186:1153–7.CrossRefGoogle Scholar
  42. 42.
    Fielding S, Ogbuagu A, Sivasubramaniam S, MacLennan G, Ramsay CR. Reporting and dealing with missing quality of life data in RCTs: has the picture changed in the last decade? Qual Life Res. 2016;25:2977–83.CrossRefGoogle Scholar
  43. 43.
    Wood AM, White IR, Thompson SG. Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clin Trials. 2004;1:368–76.CrossRefGoogle Scholar
  44. 44.
    Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ. 2015;350:h391.CrossRefGoogle Scholar
  45. 45.
    Hewitt CE, Kumaravel B, Dumville JC, Torgerson DJ, Trial attrition study group. Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010;63:1264–70.CrossRefGoogle Scholar
  46. 46.
    Jarvinen TL, Sihvonen R, Bhandari M, Sprague S, Malmivaara A, Paavola M, et al. Blinded interpretation of study results can feasibly and effectively diminish interpretation bias. J Clin Epidemiol. 2014;67:769–72.CrossRefGoogle Scholar
  47. 47.
    Armijo-Olivo S, Ospina M, da Costa BR, Egger M, Saltaji H, Fuentes J, et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS ONE. 2014;9:e96920.CrossRefGoogle Scholar
  48. 48.
    Greenland S, Schlesselman JJ, Criqui MH. The fallacy of employing standardized regression coefficients and correlations as measures of effect. Am J Epidemiol. 1986;123:203–8.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Pharmaceutical Sciences Postgraduate ProgrammeFederal University of ParanáCuritibaBrazil
  2. 2.Hospital de ClínicasFederal University of ParanáCuritibaBrazil
  3. 3.Department of PharmacyFederal University of ParanáCuritibaBrazil
  4. 4.Research Institute for Medicines (iMed.ULisboa), Faculty of PharmacyUniversity of LisbonLisbonPortugal

Personalised recommendations