An Evidence-Hierarchical Decision Aid for Ranking in Evidence-Based Medicine

  • Jürgen LandesEmail author
Part of the Boston Studies in the Philosophy and History of Science book series (BSPS, volume 338)


This chapter addresses the problem of ranking available drugs in guideline development to support clinicians in their work. Based on a pragmatic approach to the notion of evidence and a hierarchical view on different kinds of evidence this chapter introduces a decision aid, HiDAD, which draws on the multi criteria decision making literature. This decision aid implements the wide-spread intuition that there are different kinds of evidence with varying degrees of importance by relying on a strict ordinal ordering of kinds of evidence. In order to construct a ranking every pair of drugs is first compared separately on all kinds of evidence. Next, these quantitative comparisons are then aggregated into an overall comparison between drugs based on all the available evidence in a way which avoids that evidence of less importance is trumped by evidence of the higher levels. Finally, these overall comparisons are used to determine the final ranking of drugs which then informs the process of guideline writing. Properties, modifications and applicability of the decision aid HiDAD are discussed and assessed.



The idea for this chapter arose when the author was a research assistant on a project on the “Optimal design of biofuel production by microalgae” at INRA, UR0050, Laboratoire de Biotechnologie de l’Environnement (2009–2010). Progress all but ceased when the author joined the “From objective Bayesian epistemology to inductive logic” AHRC-funded project at the University of Kent. The great majority of the work was carried out after the author joined the ERC-funded project “Philosophy of Pharmacology: Safety, Statistical Standards, and Evidence Amalgamation” (grant 639276) at the LMU Munich. Currently, the author is the principal investigator of the project Evidence and Objective Bayesian Epistemology funded by the German Research Council. Regarding this chapter, the author benefited from a number of discussions with Seamus Bradley, Ricardo Büttner, Teddy Groves, Adam LaCaze, Laurent Lardon, Barbara Osimani, Roland Poellinger, David Teira and Jon Williamson as well as the members of the Environmental Lifecycle and Sustainability Assessment group. He would also like to thank an anonymous referee for the European Journal of Operational Research and three anonymous reviewers for this volume as well as the editors of this volume for their thoughtful comments and insights.


  1. Alonso-Coello, P., Schünemann, H. J., Moberg, J., Brignardello-Petersen, R., Akl, E. A., Davoli, M., Treweek, S., Mustafa, R. A., Rada, G., Rosenbaum, S., Morelli, A., Guyatt, G. H., & Oxman, A. D. (2016). GRADE Evidence to Decision (EtD) frameworks: A systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ, 353.
  2. Aumann, R. J. (1962). Utility theory without the completeness axiom. Econometrica, 30(3), 445–462.CrossRefGoogle Scholar
  3. Belton, V. (2009). Matters of weight. Journal of Multi-Criteria Decision Analysis, 15, 109–110.CrossRefGoogle Scholar
  4. Belton, V., & Stewart, T. J. (2002). Multiple criteria decision analysis: An integrated approach. Boston: Springer.CrossRefGoogle Scholar
  5. Bertamini, M., & Munafó, M. R. (2012). Bite-size science and its undesired side effects. Perspectives on Psychological Science, 7(1), 67–71.CrossRefGoogle Scholar
  6. Bes-Rastrollo, M., Schulze, M. B., Ruiz-Canela, M., & Martinez-Gonzalez, M. A. (2013). Financial conflicts of interest and reporting bias regarding the association between sugar-sweetened beverages and weight gain: A systematic review of systematic reviews. PLoS Medicine, 10(12), 1–9.CrossRefGoogle Scholar
  7. Billaut, J.-C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? An MCDM view. Scientometrics, 84, 237–263.CrossRefGoogle Scholar
  8. Borm, G. F., Lemmers, O., Fransen, J., & Donders, R. (2009). The evidence provided by a single trial is less reliable than its statistical analysis suggests. Journal of Clinical Epidemiology, 62(7), 711–715.CrossRefGoogle Scholar
  9. Boström, H., Andler, S. F., Brohede, M., & Johansson, R. (2007). On the definition of information fusion as a field of research. Technical report, University of Skövde.Google Scholar
  10. Boutilier, C. (1994). Toward a logic for qualitative decision theory. In Proceedings of KR (Vol. 94, pp. 75–86). San Fransisco: Morgan Kaufmann.Google Scholar
  11. Bouyssou, D., Marchant, T., Pirlot, M., Tsoukiàs, A., & Vincke, P. (2006). Evaluation and decision models with multiple criteria. New York: Springer.Google Scholar
  12. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafo, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.CrossRefGoogle Scholar
  13. Canadian Task Force on the Periodic Health Examination. (1979). The periodic health examination. Canadian Medical Association Journal, 121(9), 1193–1254.Google Scholar
  14. Carnap, R. (1947). On the application of inductive logic. Philosophy and Phenomenological Research, 8(1), 133–148.CrossRefGoogle Scholar
  15. Cartwright, N. (2007). Are RCTs the gold standard? Biosocieties, 2(1), 11–20.CrossRefGoogle Scholar
  16. Cartwright, N., & Munro, E. (2010). The limitations of randomized controlled trials in predicting effectiveness. Journal of Evaluation in Clinical Practice, 16(2), 260–266.CrossRefGoogle Scholar
  17. Chalmers, I., Hedges, L. V., & Cooper, H. (2002). A brief history of research synthesis. Evaluation & the Health Professions, 25(1), 12–37.CrossRefGoogle Scholar
  18. Chan, A.-W., & Altman, D. G. (2005). Epidemiology and reporting of randomised trials published in PubMed journals. The Lancet, 365(9465), 1159–1162.CrossRefGoogle Scholar
  19. Clarke, B., Gillies, D., Illari, P., Russo, F., & Williamson, J. (2013). The evidence that evidence-based medicine omits. Preventive Medicine, 57(6), 745–747.CrossRefGoogle Scholar
  20. Clarke, B., Gillies, D., Illari, P., Russo, F., & Williamson, J. (2014). Mechanisms and the evidence hierarchy. Topoi, 33, 339–360.CrossRefGoogle Scholar
  21. Cooper, N., Coyle, D., Abrams, K., Mugford, M., & Sutton, A. (2005). Use of evidence in decision models: An appraisal of health technology assessments in the UK since 1997. Journal of Health Services Research & Policy, 10(4), 245–250.CrossRefGoogle Scholar
  22. Doll, R., & Peto, R. (1980). Randomised controlled trials and retrospective controls. British Medical Journal, 280, 44.CrossRefGoogle Scholar
  23. Doyle, J., & Thomason, R. H. (1999). Background to qualitative decision theory. AI Magazine, 20(2), 55–68.Google Scholar
  24. Dubois, D., Fargier, H., & Perny, P. (2002). On the limitations of ordinal approaches to decision-making. In D. Fensel, F. Giunchiglia, D. L. McGuinness, & M.-A. Williams (Eds.), Proceedings of KR (pp. 133–146). San Fransisco: Morgan Kaufmann.Google Scholar
  25. Dubois, D., Fargier, H., & Prade, H. (1997). Decision-making under ordinal preferences and comparative uncertainty. In Proceeding of UAI (pp. 157–164).Google Scholar
  26. Dubois, D., Fargier, H., Prade, H., & Sabadin, R. (2009). A survey of qualitative decision rules under uncertainty. In Decision-making Process (Chapter 11, pp. 435–473). London: Wiley Online Library.Google Scholar
  27. Earman, J. (1992). Bayes or bust? MIT. Cambridge, MAGoogle Scholar
  28. Etz, A., & Vandekerckhove, J. (2016). A bayesian perspective on the reproducibility project: Psychology. PLoS ONE, 11(2).Google Scholar
  29. Evans, D. (2003). Hierarchy of evidence: A framework for ranking evidence evaluating healthcare interventions. Journal of Clinical Nursing, 12(1), 77–84.CrossRefGoogle Scholar
  30. Every-Palmer, S., & Howick, J. (2014). How evidence-based medicine is failing due to biased trials and selective publication. Journal of Evaluation in Clinical Practice, 20(6), 908–914.CrossRefGoogle Scholar
  31. Figueira, J., Greco, S., & Ehrgott, M. (2005). Multiple criteria decision analysis: State of the art surveys. Springer.CrossRefGoogle Scholar
  32. Gigerenzer, G., & Gaissmaier, W. (2011). Heuristic decision making. Annual Review of Psychology, 62(1), 451–482.CrossRefGoogle Scholar
  33. GRADE Working Group (2004). Grading quality of evidence and strength of recommendations. British Medical Journal, 328(7454), 1490–1494.CrossRefGoogle Scholar
  34. Guyatt, G., Oxman, A. D., Sultan, S., Brozek, J., Glasziou, P., Alonso-Coello, P., Atkins, D., Kunz, R., Montori, V., Jaeschke, R., Rind, D., Dahm, P., Akl, E. A., Meerpohl, J., Vist, G., Berliner, E., Norris, S., Falck-Ytter, Y., & Schünemann, H. J. (2013). GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. Journal of Clinical Epidemiology, 66(2), 151–157.CrossRefGoogle Scholar
  35. Guyatt, G. H., Oxman, A. D., Schünemann, H. J., Tugwell, P., & Knottnerus, A. (2011). GRADE guidelines: A new series of articles in the journal of clinical epidemiology. Journal of Clinical Epidemiology, 64, 380–382.CrossRefGoogle Scholar
  36. Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. British Medical Journal, 336(7650), 924–926.CrossRefGoogle Scholar
  37. Horton, R. (2004). Vioxx, the implosion of Merck, and aftershocks at the FDA. The Lancet, 364(9450), 1995–1996.CrossRefGoogle Scholar
  38. Howick, J. H. (2011). The philosophy of evidence-based medicine. Chichester: Blackwell.CrossRefGoogle Scholar
  39. Howick, J., et al. (2011). The Oxford 2011 levels of evidence. Oxford: Oxford Centre for Evidence Based Medicine. Scholar
  40. Jüni, P., Nartey, L., Reichenbach, S., Sterchi, R., Dieppe, P. A., & Egger, M. (2004). Risk of cardiovascular events and rofecoxib: Cumulative meta-analysis. The Lancet, 364(9450), 2021–2029.CrossRefGoogle Scholar
  41. Keeney, R. L., & Raiffa, H. (1993). Decisions with multiple objectives. Cambridge: Cambridge Books. Cambridge University Press.CrossRefGoogle Scholar
  42. Kelly, M. P., & Moore, T. A. (2012). The judgement process in evidence-based medicine and health technology assessment. Social Theory & Health, 10(1), 1–19.CrossRefGoogle Scholar
  43. Kelly, T. (2015). Evidence. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy (Fall 2014 ed.). Stanford: Metaphysics Research Lab, Stanford University. Google Scholar
  44. Khangura, S., Polisena, J., Clifford, T. J., Farrah, K., & Kamel, C. (2014). Rapid review: An emerging approach to evidence synthesis in health technology assessment. International Journal of Technology Assessment in Health Care, 30(1), 1–8.CrossRefGoogle Scholar
  45. Krumholz, H. M., Ross, J. S., Presler, A. H., & Egilman, D. S. (2007). What have we learnt from vioxx? British Medical Journal, 334(7585), 120–123.CrossRefGoogle Scholar
  46. LaCaze, A. (2009). Evidence-based medicine must be …. Journal of Medicine and Philosophy, 34(5), 509–527.CrossRefGoogle Scholar
  47. Landes, J., Osimani, B., & Poellinger, R. (2018). Epistemology of causal inference in pharmacology. European Journal for Philosophy of Science, 8, 3–49.CrossRefGoogle Scholar
  48. Lehtinen, A. (2013). On the impossibility of amalgamating evidence. Journal for General Philosophy of Science, 44(1), 101–110.CrossRefGoogle Scholar
  49. Marewski, J. N., & Gigerenzer, G. (2012). Heuristic decision making in medicine. Dialogues in Clinical Neuroscience, 14, 77–89.Google Scholar
  50. McGauran, N., Wieseler, B., Kreis, J., Schuler, Y.-B., Kolsch, H., & Kaiser, T. (2010). Reporting bias in medical research – A narrative review. Trials, 11(37), 1–15.Google Scholar
  51. Mussen, F., Salek, S., & Walker, S. (2009). Benefit-risk appraisal of medicines. Chichester: Wiley.Google Scholar
  52. Onakpoya, I. J., Heneghan, C. J., & Aronson, J. K. (2016). Worldwide withdrawal of medicinal products because of adverse drug reactions: A systematic review and analysis. Critical Reviews in Toxicology, 46, 477–489.CrossRefGoogle Scholar
  53. Osimani, B. (2014a). Hunting side effects and explaining them: Should we reverse evidence hierarchies upside down? Topoi, 33(2), 295–312.CrossRefGoogle Scholar
  54. Osimani, B. (2014b). Safety vs. efficacy assessment of pharmaceuticals: Epistemological rationales and methods. Preventive Medicine Reports, 1, 9–13.CrossRefGoogle Scholar
  55. Price, K. L., Amy Xia, H., Lakshminarayanan, M., Madigan, D., Manner, D., Scott, J., Stamey, J. D., & Thompson, L. (2014). Bayesian methods for design and analysis of safety trials. Pharmaceutical Statistics, 13(1), 13–24.CrossRefGoogle Scholar
  56. Reiss, J. (2015). A pragmatist theory of evidence. Philosophy of Science, 82(3), 341–362.CrossRefGoogle Scholar
  57. Revicki, D. A., & Frank, L. (1999). Pharmacoeconomic evaluation in the real world. PharmacoEconomics, 15(5), 423–434.CrossRefGoogle Scholar
  58. Russo, F., & Williamson, J. (2007). Interpreting causality in the health sciences. International Studies in the Philosophy of Science, 21(2), 157–170.CrossRefGoogle Scholar
  59. Sackett, D. L., Straus, S. E., Richardson, W. S., & Haynes, R. B. (2000). Evidence-based medicine: How to practice and teach EBM (2nd ed.). New York: Churchill Livingstone.Google Scholar
  60. Savage, L. J. (1954). The foundations of statistics. New York: Dover Publications.Google Scholar
  61. Shafer, G. (1976). A mathematical theory of evidence. Princeton: Princeton University Press.Google Scholar
  62. Shafer, G., & Srivastava, R. (1990). The bayesian and belief-function formalisms: A general perspective for auditing. Auditing: A Journal of Practice & Theory, 9, 110–137.Google Scholar
  63. Skyrms, B. (1990). The dynamics of rational deliberation. Cambridge, MA: Harvard University Press.Google Scholar
  64. Solomon, M. (2011). Just a paradigm: Evidence-based medicine in epistemological context. European Journal for Philosophy of Science, 1(3), 451–466.CrossRefGoogle Scholar
  65. Stegenga, J. (2013). An impossibility theorem for amalgamating evidence. Synthese, 190(12), 2391–2411.CrossRefGoogle Scholar
  66. Stegenga, J. (2014). Down with the hierarchies. Topoi, 33(2), 313–322.CrossRefGoogle Scholar
  67. Tan, S.-W., & Pearl, J. (1994). Qualitative decision theory. In Proceedings of AAAI (vol. 2, pp. 928–933). Palo Alto: AAAI Press.Google Scholar
  68. Tervonen, T., van Valkenhoef, G., Buskens, E., Hillege, H. L., & Postmus, D. (2011). A stochastic multicriteria model for evidence-based decision making in drug benefit-risk analysis. Statistics in Medicine, 30(12), 1419–1428.CrossRefGoogle Scholar
  69. Thomas, J., Newman, M., & Oliver, S. (2013). Rapid evidence assessments of research to inform social policy: taking stock and moving forward. Evidence & Policy: A Journal of Research, Debate and Practice, 9(1), 5–27.CrossRefGoogle Scholar
  70. Troffaes, M. C. M., & de Cooman, G. (2014). Lower previsions. Chichester: Wiley.CrossRefGoogle Scholar
  71. Twinning, W. (2011). Moving beyond law: Interdisciplinarity and the study of evidence. In P. Dawid, W. Twinning, & M. Vasilaki (Eds.), Evidence, inference and enquiry (chapter 4, pp. 73–118). Oxford University Press.Google Scholar
  72. Upshur, R. (1995). Looking for rules in a world of exceptions: Reflections on evidence-based practice. Perspectives in Biology and Medicine, 48(4), 477–489.CrossRefGoogle Scholar
  73. Urfalino, P. (2012). Reasons and preferences in medicine evaluation committees. In H. Landemore & J. Elster (Eds.), Collective wisdom (pp. 173–202). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  74. van Valkenhoef, G., Tervonen, T., Zwinkels, T., de Brock, B., & Hillege, H. (2013). ADDIS: A decision support system for evidence-based medicine. Decision Support Systems, 55(2), 459–475.CrossRefGoogle Scholar
  75. Vandenbroucke, J. P. (2008). Observational research, randomised trials, and two views of medical science. PLoS Medicine, 5(3), e67.CrossRefGoogle Scholar
  76. Williamson, J. (2015). Deliberation, judgement and the nature of evidence. Economics and Philosophy, 31, 27–65.CrossRefGoogle Scholar
  77. Worrall, J. (2002). What evidence in evidence-based medicine? Philosophy of Science, 69(3), 316–330.CrossRefGoogle Scholar
  78. Worrall, J. (2007a). Evidence in medicine and evidence-based medicine. Philosophy Compass, 2(6), 981–1022.CrossRefGoogle Scholar
  79. Worrall, J. (2007b). Why there’s no cause to randomize. British Journal for the Philosophy of Science, 58(3), 451–488.CrossRefGoogle Scholar
  80. Worrall, J. (2010). Evidence: Philosophy of science meets medicine. Journal of Evaluation in Clinical Practice, 16(2), 356–362.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Munich Center for Mathematical PhilosophyLMU MunichMunichGermany

Personalised recommendations