Skip to main content
SpringerLink
Log in

Machine Learning in Evidence Synthesis Research

  • Chapter
  • First Online:
Machine Learning in Dentistry

  • 997 Accesses

Abstract

In this chapter, we will explore how Systemic Reviews (SR) are traditionally conducted and how the process of arriving at a valuable SR can be made more efficient and less error prone using Machine Learning (ML) techniques. As the integration of ML at the screening stage of SRs has reached the highest level of maturity, we will explain the techniques utilized. We will further describe the extraction process from primary studies supported by ML techniques. The discussion of pitfalls when conducting SRs concludes the chapter, specifically how ML can address bias. Lastly, we address the inherent limitations of artificial intelligence in healthcare with a special emphasis on ML for the use in SRs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 139.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Brignardello-Petersen R, Carrasco-Labra A, Glick M, Guyatt GH, Azarpazhooh A. A practical approach to evidence-based dentistry: understanding and applying the principles of EBD. J Am Dent Assoc. 2014;145:1105–7. https://doi.org/10.14219/jada.2014.102.

    Article  PubMed  Google Scholar 

  2. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312:71–2.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Chandler J, Cumpston M, Thomas J, Higgins JP, Deeks JJ, Clarke MJ. Chapter 1: introduction. In: Higgins JPT, Thomas J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated august 2019); 2019. Cochrane.

    Google Scholar 

  4. Read K, Creamer A, Kafel D, Vander Hart RJ, Martin ER. Building an escience thesaurus for librarians: a collaboration between the National Network of libraries of medicine, New England region and an associate fellow at the National Library of medicine. J eSci Librariansh. 2013;2:53–67.

    Article  Google Scholar 

  5. Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, Perrier L, Hutton B, Moher D, Straus SE. A scoping review of rapid review methods. BMC Med. 2015;13:244. https://doi.org/10.1186/s12916-015-0465-6.

    Article  Google Scholar 

  6. Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods. 2014;5:371–85. https://doi.org/10.1002/jrsm.1123.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10(Suppl 1):21–34. https://doi.org/10.1258/1355819054308530.

    Article  PubMed  Google Scholar 

  8. Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7:e012545. https://doi.org/10.1136/bmjopen-2016-012545.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7:e1000326. https://doi.org/10.1371/journal.pmed.1000326.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc. 2006;13:206–19. https://doi.org/10.1197/jamia.M1929.

    Article  PubMed  PubMed Central  Google Scholar 

  11. O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:5. https://doi.org/10.1186/2046-4053-4-5.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Marshall CM. Sutton A. SR tool box [WWW Document]. 2019.

    Google Scholar 

  13. Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S. Chapter 2: determining the scope of the review and the questions it will address. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook.

  14. Kugley S, Wade A, Thomas J, Mahood Q, Klint-Jorgensen AM, Hammerstrom K, Sathe N. A guide to information retrieval for cambell systematic reviews. 2010.

    Google Scholar 

  15. Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014;3:74. https://doi.org/10.1186/2046-4053-3-74.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Cohen AM, Smalheiser NR, McDonagh MS, Yu C, Adams CE, Davis JM, Yu PS. Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine. J Am Med Inform Assoc. 2015;22:707–17. https://doi.org/10.1093/jamia/ocu025.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Marshall IJ, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide. Res Synth Methods. 2018;9:602–14. https://doi.org/10.1002/jrsm.1287.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Wallace BC, Noel-Storr A, Marshall IJ, Cohen AM, Smalheiser NR, Thomas J. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc. 2017;24:1165–8. https://doi.org/10.1093/jamia/ocx053.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Cochrane Crowd. 2019.

    Google Scholar 

  20. Glanville J, Lefebvre C, Wright K. ISSG search filter resource York (UK): the InterTASC information specialists’ sub-group [WWW document]. 2019. https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home. Accessed Sep 10 2019.

  21. O’Connor AM, Tsafnat G, Gilbert SB, Thayer KA, Shemilt I, Thomas J, Glasziou P, Wolfe MS. Still moving toward automation of the systematic review process: a summary of discussions at the third meeting of the international collaboration for automation of systematic reviews (ICASR). Syst Rev. 2019;8:57. https://doi.org/10.1186/s13643-019-0975-y.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Levay P, Craven J. Systematic searching- chapter 7. London: Facet Publishing; 2019.

    Google Scholar 

  23. Introduction what is MeSH? [WWW Document]. 2019. https://www.nlm.nih.gov/bsd/disted/meshtutorial/introduction/index.html Accessed Sep 10 2019.

  24. Soto AJ, Przybyla P, Ananiadou S. Thalia: semantic search engine for biomedical abstracts. Bioinformatics. 2018;35:1799–801.

    Article  PubMed Central  Google Scholar 

  25. Incorporating values for indexing method in MEDLINE/PubMed CML [WWW Document]. 2018.

    Google Scholar 

  26. Jimeno-Yepes A, Mork JG, Wilkowski B, Fushman DD, Aronson AR. MEDLINE MeSH indexing: lessons learned from machine learning and future directions. 2012.

    Google Scholar 

  27. Mork J, Aronson A, Demner-Fushman D. 12 years on - is the NLM medical text indexer still useful and relevant? J Biomed Semantics. 2017;8:8. https://doi.org/10.1186/s13326-017-0113-5.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinform. 2010b;11:55. https://doi.org/10.1186/1471-2105-11-55.

    Article  Google Scholar 

  29. Urquhart O, Tampi MP, Pilcher L, Slayton RL, Araujo MWB, Fontana M, Guzmán-Armstrong S, Nascimento MM, Nový BB, Tinanoff N, Weyant RJ, Wolff MS, Young DA, Zero DT, Brignardello-Petersen R, Banfield L, Parikh A, Joshi G, Carrasco-Labra A. Nonrestorative treatments for caries: systematic review and network meta-analysis. J Dent Res. 2019;98:14–26. https://doi.org/10.1177/0022034518800014.

    Article  PubMed  Google Scholar 

  30. Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst Rev. 2019;8:163. https://doi.org/10.1186/s13643-019-1074-9.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Przybyła P, Brockmeier AJ, Kontonatsios G, Pogam M-AL, McNaught J, von Elm E, Nolan K, Ananiadou S. Prioritising references for systematic reviews with RobotAnalyst: a user study. Res Synth Methods. 2018;9:470–88. https://doi.org/10.1002/jrsm.1311.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Thomas J, Brunton J, Graziosi S. EPPI-reviewer 4: software for research synthesis. London: Social Science Research Unit, UCL Institute of Education; 2010.

    Google Scholar 

  33. Wallace BC, Small K, Brodley C, Lau J, Trikalinos TA. Modeling annotation time to reduce workload in Comparative Effectivenss reviews. 2010a.

    Google Scholar 

  34. Howard BE, Phillips J, Miller K, Tandon A, Mav D, Shah MR, Holmgren S, Pelch KE, Walker V, Rooney AA, Macleod M, Shah RR, Thayer K. SWIFT-review: a text-mining workbench for systematic review. Syst Rev. 2016;5:87. https://doi.org/10.1186/s13643-016-0263-z.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Wallace BC, Small K, Brodley CE, Lau J, Schmid CH, Bertram L, Lill CM, Cohen JT, Trikalinos TA. Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining. Genet Med. 2012a;14:663–9. https://doi.org/10.1038/gim.2012.7.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In Proceedings of the 2nd ACM SIGHIT International Health Informatics. 2012b.

    Google Scholar 

  37. Yu W, Clyne M, Dolan SM, Yesupriya A, Wulf A, Liu T, Khoury MJ, Gwinn M. GAPscreener: an automatic tool for screening human genetic association literature in PubMed using the support vector machine technique. BMC Bioinform. 2008;9:205. https://doi.org/10.1186/1471-2105-9-205.

    Article  Google Scholar 

  38. DistillerSR [WWW Document]. Systematic review and literature review software by evidence partners. 2019. https://www.evidencepartners.com/products/distillersr-systematic-review-software/. Accessed Oct 10 2019.

  39. Shemilt I, Simon A, Hollands GJ, Marteau TM, Ogilvie D, O’Mara-Eves A, Kelly MP, Thomas J. Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Res Synth Methods. 2014;5:31–49. https://doi.org/10.1002/jrsm.1093.

    Article  PubMed  Google Scholar 

  40. Olorisade BK, Brereton P, Andras P. Reproducibility of studies on text mining for citation screening in systematic reviews: evaluation and checklist. J Biomed Inform. 2017;73:1–13. https://doi.org/10.1016/j.jbi.2017.07.010.

    Article  PubMed  Google Scholar 

  41. Reeves BC, Deeks JJ, Higgins JPT, Shea B, Tugwell P, Wells GA. Chapter 24: Including non-randomized studies on intervention effects. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane. 2019.

    Google Scholar 

  42. Matwin S, Kouznetsov A, Inkpen D, Frunza O, O’Blenis P. A new algorithm for reducing the workload of experts in performing systematic reviews. J Am Med Inform Assoc. 2010;17:446–53. https://doi.org/10.1136/jamia.2010.004325.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Cohen AM. Optimizing feature representation for automated systematic review work prioritization. AMIA Annu Symp Proc. 2008;2008:121–5.

    PubMed Central  Google Scholar 

  44. Cohen AM, Ambert K, McDonagh M. Studying the potential impact of automated document classification on scheduling a systematic review update. BMC Med Inform Decis Mak. 2012;12:33. https://doi.org/10.1186/1472-6947-12-33.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Cohen AM, Ambert K, McDonagh M. Cross-topic learning for work prioritization in systematic review creation and update. J Am Med Inform Assoc. 2009;16:690–704. https://doi.org/10.1197/jamia.M3162.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Wallace BC, Small K, Brodley CE, Trikalinos TA. Who should label what? Instance allocation in multiple expert active learning. SDM. 2011. https://doi.org/10.1137/1.9781611972818.16.

  47. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med. 2010;152:726–32. https://doi.org/10.7326/0003-4819-152-11-201006010-00232.

    Article  PubMed  Google Scholar 

  48. Al-Namankany AA, Ashley P, Moles DR, Parekh S. Assessment of the quality of reporting of randomized clinical trials in paediatric dentistry journals. Int J Paediatr Dent. 2009;19:318–24. https://doi.org/10.1111/j.1365-263X.2009.00974.x.

    Article  PubMed  Google Scholar 

  49. Hua F, Deng L, Kau CH, Jiang H, He H, Walsh T. Reporting quality of randomized controlled trial abstracts: survey of leading general dental journals. J Am Dent Assoc. 2015;146:669–678.e1. https://doi.org/10.1016/j.adaj.2015.03.020.

    Article  PubMed  Google Scholar 

  50. Kloukos D, Papageorgiou SN, Doulis I, Petridis H, Pandis N. Reporting quality of randomised controlled trials published in prosthodontic and implantology journals. J Oral Rehabil. 2015;42:914–25. https://doi.org/10.1111/joor.12325.

    Article  PubMed  Google Scholar 

  51. Lempesi E, Koletsi D, Fleming PS, Pandis N. The reporting quality of randomized controlled trials in orthodontics. J Evid Based Dent Pract. 2014;14:46–52. https://doi.org/10.1016/j.jebdp.2013.12.001.

    Article  PubMed  Google Scholar 

  52. Sarkis-Onofre R, Poletto-Neto V, Cenci MS, Pereira-Cenci T, Moher D. Impact of the CONSORT statement endorsement in the completeness of reporting of randomized clinical trials in restorative dentistry. J Dent. 2017;58:54–9. https://doi.org/10.1016/j.jdent.2017.01.009.

    Article  PubMed  Google Scholar 

  53. Savithra P, Nagesh LS. Have CONSORT guidelines improved the quality of reporting of randomised controlled trials published in public health dentistry journals? Oral Health Prev Dent. 2013;11:95–103. https://doi.org/10.3290/j.ohpd.a29359.

    Article  PubMed  Google Scholar 

  54. Li T, Higgins JPT, Deeks JJ. Chapter 5: Collecting data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook.

  55. Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Syst Rev. 2015;4:78. https://doi.org/10.1186/s13643-015-0066-7.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Kiritchenko S, de Bruijn B, Carini S, Martin J, Sim I. ExaCT: automatic extraction of clinical trial characteristics from journal publications. BMC Med Inform Decis Mak. 2010;10:56. https://doi.org/10.1186/1472-6947-10-56.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, Als-Nielsen B, Balk EM, Gluud C, Gluud LL, Ioannidis JP, Schulz KF, Beynon R, Welton NJ, Wood L, Moher D, Deeks JJ, Sterne JA. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012;157:429–38. https://doi.org/10.7326/0003-4819-157-6-201209180-00537.

    Article  PubMed  Google Scholar 

  58. Guyatt G, Rennier D, Meade MO, Cook DJ. Glossary. In: Guyatt G, Rennier D, Meade MO, Cook DJ, editors. Users’ guide to the medical literature: a manual for evidence-based clinical practice. New York: McGraw-Hill Education; 2015. p. 645.

    Google Scholar 

  59. Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook.

  60. Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook.

  61. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JAC. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;342:d5928.

    Article  Google Scholar 

  62. Hartling L, Ospina M, Liang Y. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012.

    Article  PubMed  PubMed Central  Google Scholar 

  63. Millard LAC, Flach PA, Higgins JPT. Machine learning to assist risk-of-bias assessments in systematic reviews. Int J Epidemiol. 2016;45:266–77. https://doi.org/10.1093/ije/dyv306.

    Article  PubMed  Google Scholar 

  64. Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc. 2016;23:193–201. https://doi.org/10.1093/jamia/ocv044.

    Article  PubMed  Google Scholar 

  65. Soboczenski F, Trikalinos TA, Kuiper J, Bias RG, Wallace BC, Marshall IJ. Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study. BMC Med Inform Decis Mak. 2019;19:96. https://doi.org/10.1186/s12911-019-0814-z.

    Article  PubMed  PubMed Central  Google Scholar 

  66. Sauro J. Measuring usability with the system usability scale (SUS). 2011. http://www.measuringusability.com/sus.php [WWW document].

  67. Nordling L. Mind the gap. Nature. 2019;573:S103–5.

    Article  PubMed  Google Scholar 

  68. Dawson D, Schleiger E, Horton J, McLaughlin J, Robinson C, Quesada G, Scowcroft J, Hajkowicz S. Artificial intelligence: Australia’s ethics framework. Data61 CSIRO, Australia. 2019.

    Google Scholar 

  69. JASON, The MITRE Corporation. Artificial intelligence for health and health care. JSR-17-Task-002. 2017.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Evidence Synthesis and Translation Research, Science Institute, American Dental Association, Chicago, IL, USA

    Alonso Carrasco-Labra

  2. Department of Oral and Craniofacial Health Science, School of Dentistry, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Alonso Carrasco-Labra

  3. Health Research Analyst, Department of Evidence Synthesis and Translation Research, Science Institute, American Dental Association, Chicago, IL, USA

    Olivia Urquhart

  4. Head of the School and Dean, The University of Sydney School of Dentistry, Westmead, NSW, Australia

    Heiko Spallek

  5. Academic Lead Digital Health and Health Service Informatics, Faculty of Medicine and Health, The University of Sydney, Westmead, NSW, Australia

    Heiko Spallek

Authors
  1. Alonso Carrasco-Labra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Olivia Urquhart
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Heiko Spallek
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alonso Carrasco-Labra .

Editor information

Editors and Affiliations

  1. Ohio State University, Columbus, OH, USA

    Ching-Chang Ko

  2. School of Biomedical Engineering, ShanghaiTech University, Shanghai, Shanghai, China

    Dinggang Shen

  3. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Li Wang

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Carrasco-Labra, A., Urquhart, O., Spallek, H. (2021). Machine Learning in Evidence Synthesis Research. In: Ko, CC., Shen, D., Wang, L. (eds) Machine Learning in Dentistry. Springer, Cham. https://doi.org/10.1007/978-3-030-71881-7_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-71881-7_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-71880-0

  • Online ISBN: 978-3-030-71881-7

  • eBook Packages: MedicineMedicine (R0)

Publish with us

Policies and ethics

Search

Navigation