Skip to main content

Machine Learning in Evidence Synthesis Research

  • 440 Accesses

Abstract

In this chapter, we will explore how Systemic Reviews (SR) are traditionally conducted and how the process of arriving at a valuable SR can be made more efficient and less error prone using Machine Learning (ML) techniques. As the integration of ML at the screening stage of SRs has reached the highest level of maturity, we will explain the techniques utilized. We will further describe the extraction process from primary studies supported by ML techniques. The discussion of pitfalls when conducting SRs concludes the chapter, specifically how ML can address bias. Lastly, we address the inherent limitations of artificial intelligence in healthcare with a special emphasis on ML for the use in SRs.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-71881-7_12
  • Chapter length: 15 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   119.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-71881-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)
Hardcover Book
USD   159.99
Price excludes VAT (USA)
Fig. 12.1
Fig. 12.2
Fig. 12.3
Fig. 12.4
Fig. 12.5

References

  1. Brignardello-Petersen R, Carrasco-Labra A, Glick M, Guyatt GH, Azarpazhooh A. A practical approach to evidence-based dentistry: understanding and applying the principles of EBD. J Am Dent Assoc. 2014;145:1105–7. https://doi.org/10.14219/jada.2014.102.

    CrossRef  PubMed  Google Scholar 

  2. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312:71–2.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  3. Chandler J, Cumpston M, Thomas J, Higgins JP, Deeks JJ, Clarke MJ. Chapter 1: introduction. In: Higgins JPT, Thomas J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated august 2019); 2019. Cochrane.

    Google Scholar 

  4. Read K, Creamer A, Kafel D, Vander Hart RJ, Martin ER. Building an escience thesaurus for librarians: a collaboration between the National Network of libraries of medicine, New England region and an associate fellow at the National Library of medicine. J eSci Librariansh. 2013;2:53–67.

    CrossRef  Google Scholar 

  5. Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, Perrier L, Hutton B, Moher D, Straus SE. A scoping review of rapid review methods. BMC Med. 2015;13:244. https://doi.org/10.1186/s12916-015-0465-6.

    CrossRef  Google Scholar 

  6. Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods. 2014;5:371–85. https://doi.org/10.1002/jrsm.1123.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  7. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10(Suppl 1):21–34. https://doi.org/10.1258/1355819054308530.

    CrossRef  PubMed  Google Scholar 

  8. Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7:e012545. https://doi.org/10.1136/bmjopen-2016-012545.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  9. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7:e1000326. https://doi.org/10.1371/journal.pmed.1000326.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  10. Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc. 2006;13:206–19. https://doi.org/10.1197/jamia.M1929.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  11. O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:5. https://doi.org/10.1186/2046-4053-4-5.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  12. Marshall CM. Sutton A. SR tool box [WWW Document]. 2019.

    Google Scholar 

  13. Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S. Chapter 2: determining the scope of the review and the questions it will address. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook.

  14. Kugley S, Wade A, Thomas J, Mahood Q, Klint-Jorgensen AM, Hammerstrom K, Sathe N. A guide to information retrieval for cambell systematic reviews. 2010.

    Google Scholar 

  15. Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014;3:74. https://doi.org/10.1186/2046-4053-3-74.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  16. Cohen AM, Smalheiser NR, McDonagh MS, Yu C, Adams CE, Davis JM, Yu PS. Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine. J Am Med Inform Assoc. 2015;22:707–17. https://doi.org/10.1093/jamia/ocu025.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  17. Marshall IJ, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide. Res Synth Methods. 2018;9:602–14. https://doi.org/10.1002/jrsm.1287.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  18. Wallace BC, Noel-Storr A, Marshall IJ, Cohen AM, Smalheiser NR, Thomas J. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc. 2017;24:1165–8. https://doi.org/10.1093/jamia/ocx053.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  19. Cochrane Crowd. 2019.

    Google Scholar 

  20. Glanville J, Lefebvre C, Wright K. ISSG search filter resource York (UK): the InterTASC information specialists’ sub-group [WWW document]. 2019. https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home. Accessed Sep 10 2019.

  21. O’Connor AM, Tsafnat G, Gilbert SB, Thayer KA, Shemilt I, Thomas J, Glasziou P, Wolfe MS. Still moving toward automation of the systematic review process: a summary of discussions at the third meeting of the international collaboration for automation of systematic reviews (ICASR). Syst Rev. 2019;8:57. https://doi.org/10.1186/s13643-019-0975-y.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  22. Levay P, Craven J. Systematic searching- chapter 7. London: Facet Publishing; 2019.

    Google Scholar 

  23. Introduction what is MeSH? [WWW Document]. 2019. https://www.nlm.nih.gov/bsd/disted/meshtutorial/introduction/index.html Accessed Sep 10 2019.

  24. Soto AJ, Przybyla P, Ananiadou S. Thalia: semantic search engine for biomedical abstracts. Bioinformatics. 2018;35:1799–801.

    CrossRef  PubMed Central  Google Scholar 

  25. Incorporating values for indexing method in MEDLINE/PubMed CML [WWW Document]. 2018.

    Google Scholar 

  26. Jimeno-Yepes A, Mork JG, Wilkowski B, Fushman DD, Aronson AR. MEDLINE MeSH indexing: lessons learned from machine learning and future directions. 2012.

    Google Scholar 

  27. Mork J, Aronson A, Demner-Fushman D. 12 years on - is the NLM medical text indexer still useful and relevant? J Biomed Semantics. 2017;8:8. https://doi.org/10.1186/s13326-017-0113-5.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  28. Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinform. 2010b;11:55. https://doi.org/10.1186/1471-2105-11-55.

    CrossRef  Google Scholar 

  29. Urquhart O, Tampi MP, Pilcher L, Slayton RL, Araujo MWB, Fontana M, Guzmán-Armstrong S, Nascimento MM, Nový BB, Tinanoff N, Weyant RJ, Wolff MS, Young DA, Zero DT, Brignardello-Petersen R, Banfield L, Parikh A, Joshi G, Carrasco-Labra A. Nonrestorative treatments for caries: systematic review and network meta-analysis. J Dent Res. 2019;98:14–26. https://doi.org/10.1177/0022034518800014.

    CrossRef  PubMed  Google Scholar 

  30. Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst Rev. 2019;8:163. https://doi.org/10.1186/s13643-019-1074-9.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  31. Przybyła P, Brockmeier AJ, Kontonatsios G, Pogam M-AL, McNaught J, von Elm E, Nolan K, Ananiadou S. Prioritising references for systematic reviews with RobotAnalyst: a user study. Res Synth Methods. 2018;9:470–88. https://doi.org/10.1002/jrsm.1311.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  32. Thomas J, Brunton J, Graziosi S. EPPI-reviewer 4: software for research synthesis. London: Social Science Research Unit, UCL Institute of Education; 2010.

    Google Scholar 

  33. Wallace BC, Small K, Brodley C, Lau J, Trikalinos TA. Modeling annotation time to reduce workload in Comparative Effectivenss reviews. 2010a.

    Google Scholar 

  34. Howard BE, Phillips J, Miller K, Tandon A, Mav D, Shah MR, Holmgren S, Pelch KE, Walker V, Rooney AA, Macleod M, Shah RR, Thayer K. SWIFT-review: a text-mining workbench for systematic review. Syst Rev. 2016;5:87. https://doi.org/10.1186/s13643-016-0263-z.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  35. Wallace BC, Small K, Brodley CE, Lau J, Schmid CH, Bertram L, Lill CM, Cohen JT, Trikalinos TA. Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining. Genet Med. 2012a;14:663–9. https://doi.org/10.1038/gim.2012.7.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  36. Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In Proceedings of the 2nd ACM SIGHIT International Health Informatics. 2012b.

    Google Scholar 

  37. Yu W, Clyne M, Dolan SM, Yesupriya A, Wulf A, Liu T, Khoury MJ, Gwinn M. GAPscreener: an automatic tool for screening human genetic association literature in PubMed using the support vector machine technique. BMC Bioinform. 2008;9:205. https://doi.org/10.1186/1471-2105-9-205.

    CrossRef  Google Scholar 

  38. DistillerSR [WWW Document]. Systematic review and literature review software by evidence partners. 2019. https://www.evidencepartners.com/products/distillersr-systematic-review-software/. Accessed Oct 10 2019.

  39. Shemilt I, Simon A, Hollands GJ, Marteau TM, Ogilvie D, O’Mara-Eves A, Kelly MP, Thomas J. Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Res Synth Methods. 2014;5:31–49. https://doi.org/10.1002/jrsm.1093.

    CrossRef  PubMed  Google Scholar 

  40. Olorisade BK, Brereton P, Andras P. Reproducibility of studies on text mining for citation screening in systematic reviews: evaluation and checklist. J Biomed Inform. 2017;73:1–13. https://doi.org/10.1016/j.jbi.2017.07.010.

    CrossRef  PubMed  Google Scholar 

  41. Reeves BC, Deeks JJ, Higgins JPT, Shea B, Tugwell P, Wells GA. Chapter 24: Including non-randomized studies on intervention effects. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane. 2019.

    Google Scholar 

  42. Matwin S, Kouznetsov A, Inkpen D, Frunza O, O’Blenis P. A new algorithm for reducing the workload of experts in performing systematic reviews. J Am Med Inform Assoc. 2010;17:446–53. https://doi.org/10.1136/jamia.2010.004325.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  43. Cohen AM. Optimizing feature representation for automated systematic review work prioritization. AMIA Annu Symp Proc. 2008;2008:121–5.

    PubMed Central  Google Scholar 

  44. Cohen AM, Ambert K, McDonagh M. Studying the potential impact of automated document classification on scheduling a systematic review update. BMC Med Inform Decis Mak. 2012;12:33. https://doi.org/10.1186/1472-6947-12-33.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  45. Cohen AM, Ambert K, McDonagh M. Cross-topic learning for work prioritization in systematic review creation and update. J Am Med Inform Assoc. 2009;16:690–704. https://doi.org/10.1197/jamia.M3162.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  46. Wallace BC, Small K, Brodley CE, Trikalinos TA. Who should label what? Instance allocation in multiple expert active learning. SDM. 2011. https://doi.org/10.1137/1.9781611972818.16.

  47. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med. 2010;152:726–32. https://doi.org/10.7326/0003-4819-152-11-201006010-00232.

    CrossRef  PubMed  Google Scholar 

  48. Al-Namankany AA, Ashley P, Moles DR, Parekh S. Assessment of the quality of reporting of randomized clinical trials in paediatric dentistry journals. Int J Paediatr Dent. 2009;19:318–24. https://doi.org/10.1111/j.1365-263X.2009.00974.x.

    CrossRef  PubMed  Google Scholar 

  49. Hua F, Deng L, Kau CH, Jiang H, He H, Walsh T. Reporting quality of randomized controlled trial abstracts: survey of leading general dental journals. J Am Dent Assoc. 2015;146:669–678.e1. https://doi.org/10.1016/j.adaj.2015.03.020.

    CrossRef  PubMed  Google Scholar 

  50. Kloukos D, Papageorgiou SN, Doulis I, Petridis H, Pandis N. Reporting quality of randomised controlled trials published in prosthodontic and implantology journals. J Oral Rehabil. 2015;42:914–25. https://doi.org/10.1111/joor.12325.

    CrossRef  PubMed  Google Scholar 

  51. Lempesi E, Koletsi D, Fleming PS, Pandis N. The reporting quality of randomized controlled trials in orthodontics. J Evid Based Dent Pract. 2014;14:46–52. https://doi.org/10.1016/j.jebdp.2013.12.001.

    CrossRef  PubMed  Google Scholar 

  52. Sarkis-Onofre R, Poletto-Neto V, Cenci MS, Pereira-Cenci T, Moher D. Impact of the CONSORT statement endorsement in the completeness of reporting of randomized clinical trials in restorative dentistry. J Dent. 2017;58:54–9. https://doi.org/10.1016/j.jdent.2017.01.009.

    CrossRef  PubMed  Google Scholar 

  53. Savithra P, Nagesh LS. Have CONSORT guidelines improved the quality of reporting of randomised controlled trials published in public health dentistry journals? Oral Health Prev Dent. 2013;11:95–103. https://doi.org/10.3290/j.ohpd.a29359.

    CrossRef  PubMed  Google Scholar 

  54. Li T, Higgins JPT, Deeks JJ. Chapter 5: Collecting data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook.

  55. Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Syst Rev. 2015;4:78. https://doi.org/10.1186/s13643-015-0066-7.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  56. Kiritchenko S, de Bruijn B, Carini S, Martin J, Sim I. ExaCT: automatic extraction of clinical trial characteristics from journal publications. BMC Med Inform Decis Mak. 2010;10:56. https://doi.org/10.1186/1472-6947-10-56.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  57. Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, Als-Nielsen B, Balk EM, Gluud C, Gluud LL, Ioannidis JP, Schulz KF, Beynon R, Welton NJ, Wood L, Moher D, Deeks JJ, Sterne JA. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012;157:429–38. https://doi.org/10.7326/0003-4819-157-6-201209180-00537.

    CrossRef  PubMed  Google Scholar 

  58. Guyatt G, Rennier D, Meade MO, Cook DJ. Glossary. In: Guyatt G, Rennier D, Meade MO, Cook DJ, editors. Users’ guide to the medical literature: a manual for evidence-based clinical practice. New York: McGraw-Hill Education; 2015. p. 645.

    Google Scholar 

  59. Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook.

  60. Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook.

  61. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JAC. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;342:d5928.

    CrossRef  Google Scholar 

  62. Hartling L, Ospina M, Liang Y. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  63. Millard LAC, Flach PA, Higgins JPT. Machine learning to assist risk-of-bias assessments in systematic reviews. Int J Epidemiol. 2016;45:266–77. https://doi.org/10.1093/ije/dyv306.

    CrossRef  PubMed  Google Scholar 

  64. Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc. 2016;23:193–201. https://doi.org/10.1093/jamia/ocv044.

    CrossRef  PubMed  Google Scholar 

  65. Soboczenski F, Trikalinos TA, Kuiper J, Bias RG, Wallace BC, Marshall IJ. Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study. BMC Med Inform Decis Mak. 2019;19:96. https://doi.org/10.1186/s12911-019-0814-z.

    CrossRef  PubMed  PubMed Central  Google Scholar 

  66. Sauro J. Measuring usability with the system usability scale (SUS). 2011. http://www.measuringusability.com/sus.php [WWW document].

  67. Nordling L. Mind the gap. Nature. 2019;573:S103–5.

    CrossRef  PubMed  Google Scholar 

  68. Dawson D, Schleiger E, Horton J, McLaughlin J, Robinson C, Quesada G, Scowcroft J, Hajkowicz S. Artificial intelligence: Australia’s ethics framework. Data61 CSIRO, Australia. 2019.

    Google Scholar 

  69. JASON, The MITRE Corporation. Artificial intelligence for health and health care. JSR-17-Task-002. 2017.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Alonso Carrasco-Labra .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Carrasco-Labra, A., Urquhart, O., Spallek, H. (2021). Machine Learning in Evidence Synthesis Research. In: Ko, CC., Shen, D., Wang, L. (eds) Machine Learning in Dentistry. Springer, Cham. https://doi.org/10.1007/978-3-030-71881-7_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-71881-7_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-71880-0

  • Online ISBN: 978-3-030-71881-7

  • eBook Packages: MedicineMedicine (R0)