Protocol-Driven Searches for Medical and Health-Sciences Systematic Reviews

  • Matt-Mouley Bouamrane
  • Craig Macdonald
  • Iadh Ounis
  • Frances Mair
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6931)


Systematic reviews are instances of a critically important search task in medicine and health services research. Along with large and well conducted randomised control trials, they provide the highest levels of clinical evidence. We provide a brief overview of the methodologies used to conduct systematic reviews, and report on our recent experience of conducting a meta-review – i.e. a systematic review of reviews – of preoperative assessment. We discuss issues associated with the large manual effort currently necessary to conduct systematic reviews when using available search engines. We then suggest ways in which more dedicated and sophisticated information retrieval tools may enhance the efficiency of systematic searches and increase the recall of results. Finally, we discuss the development of tests collections for systematic reviews, to permit the development of enhanced search engines for this task.


Search Task Preoperative Assessment Test Collection Systematic Review Search Protocol 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Evans, D.: Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. Journal of Clinical Nursing 12(1), 77–84 (2003)CrossRefGoogle Scholar
  2. 2.
    NICE: The guideline development process - an overview for stakeholders, the public and the NHS (3rd ed.). National Institute for Health and Clinical Excellence (2007)Google Scholar
  3. 3.
    Doyle, J., Waters, E., Yach, D., McQueen, D., De Francisco, A., Stewart, T., Reddy, P., Gulmezoglu, A.M., Galea, G., Portela, A.: Global priority setting for Cochrane systematic reviews of health promotion and public health research. Journal of Epidemiology and Community Health 59(3), 193–197 (2005)CrossRefGoogle Scholar
  4. 4.
    Chan, A.W., Hróbjartsson, A., Haahr, M.T., Gøtzsche, P.C., Altman, D.G.: Empirical evidence for selective reporting of outcomes in randomized trials. Journal of the American Medical Association 291(20), 2457–2465 (2004)CrossRefGoogle Scholar
  5. 5.
    Sterne, J.A.C., Egger, M., Smith, G.D.: Investigating and dealing with publication and other biases in meta-analysis. BMJ 323(7304), 101–105 (2001)CrossRefGoogle Scholar
  6. 6.
    Kirkham, J.J., Dwan, K.M., Altman, D.G., Gamble, C., Dodd, S., Smyth, R., Williamson, P.R.: The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 340 (2010)Google Scholar
  7. 7.
    NHS-CRD: Centre for Reviews and Dissemination’s guidance for undertaking systematic reviews in health care. University of York (2009),
  8. 8.
    Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gotzsche, P.C., Ioannidis, J.P., Clarke, M., Devereaux, P., Kleijnen, J., Moher, D.: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of Clinical Epidemiology 62(10), e1–e34 (2009)CrossRefGoogle Scholar
  9. 9.
    Moher, D., Cook, D.J., Eastwood, S., Olkin, I., Rennie, D., Stroup, D.F.: the QUOROM Group: Improving the quality of report of meta-analyses or randomised controlled trials: the QUOROM statement. The Lancet 354, 1896–1900 (1999)CrossRefGoogle Scholar
  10. 10.
    Moher, D., Tetzlaff, J., Tricco, A.C., Sampson, M., Altman, D.G.: Epidemiology & reporting characteristics of systematic reviews. PLoS Medicine 4(3), e78 (2007)CrossRefGoogle Scholar
  11. 11.
    Liu, Y.H.: On the potential search effectiveness of MeSH (medical subject headings) terms. In: Proceedings of IIiX 2010, pp. 225–234 (2010)Google Scholar
  12. 12.
    Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G.: The PRISMA Group: Preferred reporting items for systematic reviews & meta-analyses: The PRISMA statement. PLoS Medicine 6(7) (2009)Google Scholar
  13. 13.
    World Health Organization: Safe surgery saves lives. WHO world alliance for patient safety. WHO report (2008)Google Scholar
  14. 14.
    NHS Modernisation Agency: National good practice guidance on pre-operative assessment for in patient surgery (2003)Google Scholar
  15. 15.
    Bouamrane, M.-M., Gallacher, K., Marlborough, H., Jani, B., Kinsella, J., Richards, R., van Klei, W., Mair, F.S.: Processes of preoperative assessment in elective surgery: a systematic review of reviews (2011), under reviewGoogle Scholar
  16. 16.
    Beahler, C.C., Sundheim, J.J., Trapp, N.I.: Information retrieval in systematic reviews: Challenges in public health arena. American Journal on Preventative Medicine 18, 6–10 (2000)CrossRefGoogle Scholar
  17. 17.
    Golder, S., McIntosh, H., Loke, Y.: Identifying systematic reviews of the adverse effects of health care interventions. BMC Medical Research Methodology 6(1), 22 (2006)CrossRefGoogle Scholar
  18. 18.
    Oard, D.W., Baron, J.R., Lewis, D.D.: Some lessons learned to date from the TREC Legal track (2006-2009). Technical Report, University of Maryland (2010)Google Scholar
  19. 19.
    Lupu, M., Huang, J., Zhu, J., Tait, J.: TREC-CHEM: large scale chemical information retrieval evaluation at TREC. SIGIR Forum 43, 63–70 (2009)CrossRefGoogle Scholar
  20. 20.
    Pohl, S., Zobel, J., Moffat, A.: Extended Boolean retrieval for systematic biomedical reviews. In: Proceedings of ACCS 2010, pp. 117–126 (2010)Google Scholar
  21. 21.
    Zhang, L., Ajiferuke, I., Sampson, M.: Optimizing search strategies to identify randomized controlled trials in MEDLINE. BMC Medical Research Methodology 6(23) (2006)Google Scholar
  22. 22.
    Arampatzis, A., Kamps, J., Robertson, S.: Where to stop reading a ranked list?: threshold optimization using truncated score distributions. In: Proceedings of SIGIR 2009, pp. 524–531 (2009)Google Scholar
  23. 23.
    Kwok, K.L., Grunfeld, K., Chan, M., Dinstl, N., Cool, C.: TREC-7 ad-hoc, high precision & filtering experiments using PIRCS. In: Proceedings of TREC-7 (1998)Google Scholar
  24. 24.
    Liu, T.Y.: Learning to rank for information retrieval. Foundations and Trends in Information Retrieval 3(3), 225–331 (2009)CrossRefGoogle Scholar
  25. 25.
    Roberts, P.M., Cohen, A.M., Hersh, W.R.: Tasks, topics and relevance judging for the TREC Genomics track: five years of experience evaluating biomedical text information retrieval systems. Inf. Retr. 12(1), 81–97 (2009)CrossRefGoogle Scholar
  26. 26.
    Cohen, A., Hersh, W., Peterson, K., Yen, P.Y.: Reducing workload in systematic review preparation using automated citation classification. Journal of the American Medical Informatics Association 13(2), 206–219 (2006)CrossRefGoogle Scholar
  27. 27.
    Pang, B., Lee, L.: Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval 2(1-2), 1–135 (2008)CrossRefGoogle Scholar
  28. 28.
    Macdonald, C., Santos, R.L.T., Ounis, I., Soboroff, I.: Blog track research at TREC. SIGIR Forum 44 (2010)Google Scholar
  29. 29.
    Hearst, M., Elliott, A., English, J., Sinha, R., Swearingen, K., Yee, K.P.: Finding the flow in web site search. Commun. ACM 45, 42–49 (2002)CrossRefGoogle Scholar
  30. 30.
    Boudin, F., Nie, J.Y., Dawes, M.: Deriving a test collection for clinical information retrieval from systematic reviews. In: Proceedings of DTMBIO 2010, pp. 57–60 (2010)Google Scholar
  31. 31.
    Oard, D.W., Hedin, B., Tomlinson, S., Baron, J.R.: Overview of the TREC 2008 Legal track. In: Proceedings of TREC 2008 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Matt-Mouley Bouamrane
    • 1
  • Craig Macdonald
    • 2
  • Iadh Ounis
    • 2
  • Frances Mair
    • 1
  1. 1.Institute of Health and WellbeingCollege of Medical, Veterinary and Life SciencesUK
  2. 2.School of Computing ScienceUniversity of GlasgowScotland, UK

Personalised recommendations