Skip to main content
SpringerLink
Log in

Learning-to-Rank and Relevance Feedback for Literature Appraisal in Empirical Medicine

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2018)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11018))

Abstract

The constantly expanding medical libraries contain immense amounts of information, including evidence from healthcare research. Gathering and interpreting this evidence can be both challenging and time-consuming for researchers conducting systematic reviews. Technologically assisted review (TAR) aims to assist this process by finding as much relevant information as possible with the least effort. Toward this, we present an incremental learning method that ranks documents, previously retrieved, by automating the process of title and abstract screening. Our approach combines a learning-to-rank model trained across multiple reviews with a model focused on the given review, incrementally trained based on relevance feedback. The classifiers use as features several similarity metrics between the documents and the research topic, such as Levenshtein distance, cosine similarity and BM25, and vectors derived from word embedding methods such as Word2Vec and Doc2Vec. We test our approach using the dataset provided by the Task II of CLEF eHealth 2017 and we empirically compare it with other approaches participated in the task.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.cochrane.org/.

  2. 2.

    https://www.york.ac.uk/crd/.

  3. 3.

    http://joannabriggs.org/.

  4. 4.

    http://crowd.cochrane.org/.

  5. 5.

    https://github.com/CLEF-TAR/tar.

  6. 6.

    https://github.com/CREBP/sra-polyglot.

  7. 7.

    http://scikit-learn.org/.

  8. 8.

    http://bioasq.org/.

  9. 9.

    https://github.com/CLEF-TAR/tar.

References

  1. Alharbi, A., Stevenson, M.: Ranking abstracts to identify relevant evidence for systematic reviews: the university of Sheffield’s approach to CLEF eHealth 2017 task 2: working notes for CLEF 2017. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  2. Anagnostou, A., Lagopoulos, A., Tsoumakas, G., Vlahavas, I.: Combining inter-review learning-to-rank and intra-review incremental training for title and abstract screening in systematic reviews. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  3. Bastian, H., Glasziou, P., Chalmers, I.: Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 7(9), e1000326 (2010). https://doi.org/10.1371/journal.pmed.1000326

    Article  Google Scholar 

  4. Chen, J., et al.: ECNU at 2017 eHealth task 2: technologically assisted reviews in empirical medicine. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  5. Chen, T., Guestrin, C.: XGBoost: reliable large-scale tree boosting system. arXiv, pp. 1–6 (2016). https://doi.org/10.1145/2939672.2939785

  6. Cormack, G.V., Grossman, M.R.: Technology-assisted review in empirical medicine: waterloo participation in CLEF eHealth 2017. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  7. Goeuriot, L., et al.: CLEF 2017 eHealth evaluation lab overview. In: Jones, G.J.F., et al. (eds.) CLEF 2017. LNCS, vol. 10456, pp. 291–303. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65813-1_26

    Chapter  Google Scholar 

  8. Hashimoto, K., Kontonatsios, G., Miwa, M., Ananiadou, S.: Topic detection using paragraph vectors to support active learning in systematic reviews. J. Biomed. Inform. 62, 59–65 (2016). https://doi.org/10.1016/j.jbi.2016.06.001

    Article  Google Scholar 

  9. Higgins, J.P., Green, S.: Cochrane Handbook for Systematic Reviews of Interventions. Wiley, Hoboken (2011). www.handbook.cochrane.org

    Google Scholar 

  10. Hollmann, N., Eickhoff, C.: Ranking and feedback-based stopping for recall-centric document retrieval. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  11. Howard, B.E., et al.: SWIFT-review: a text-mining workbench for systematic review. Syst. Rev. 5(1), 87 (2016). https://doi.org/10.1186/s13643-016-0263-z

    Article  Google Scholar 

  12. Kalphov, V., Georgiadis, G., Azzopardi, L.: SiS at CLEF 2017 eHealth TAR task. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  13. Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2017 technologically assisted reviews in empirical medicine overview. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  14. Kusner, M.J., Sun, Y., Kolkin, N.I., Weinberger, K.Q.: From word embeddings to document distances. In: Proceedings of the 32nd International Conference on Machine Learning, vol. 37, pp. 957–966 (2015)

    Google Scholar 

  15. Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: International Conference on Machine Learning, ICML 2014, vol. 32, pp. 1188–1196 (2014). https://doi.org/10.1145/2740908.2742760

  16. Lee, G.E.: A study of convolutional neural networks for clinical document classification in systematic reviews: sysreview at CLEF eHealth 2017. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  17. Lefebvre, C., Manheimer, E., Glanville, J.: Searching for studies. In: Cochrane Handbook for Systematic Reviews of Interventions. Cochrane Book Series, pp. 95–150 (2008). https://doi.org/10.1002/9780470712184.ch6

  18. Mikolov, T., Corrado, G., Chen, K., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of the International Conference on Learning Representations (ICLR 2013), pp. 1–12 (2013). https://doi.org/10.1162/153244303322533223

  19. Norman, C., Leeflang, M., Névéol, A.: LIMSI@CLEF eHealth 2017 task 2: logistic regression for automatic article ranking. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  20. O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., Ananiadou, S.: Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst. Rev. 4(1), 1–22 (2015). https://doi.org/10.1186/2046-4053-4-5

    Article  Google Scholar 

  21. Ouzzani, M., Hammady, H., Fedorowicz, Z., Elmagarmid, A.: Rayyan-a web and mobile app for systematic reviews. Syst. Rev. 5(1), 210 (2016). https://doi.org/10.1186/s13643-016-0384-4

    Article  Google Scholar 

  22. Qin, T., Liu, T.Y., Xu, J., Li, H.: LETOR: a benchmark collection for research on learning to rank for information retrieval. Inf. Retr. 13(4), 346–374 (2010). https://doi.org/10.1007/s10791-009-9123-y

    Article  Google Scholar 

  23. Rathbone, J., Hoffmann, T., Glasziou, P.: Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst. Rev. 4(1), 80 (2015). https://doi.org/10.1186/s13643-015-0067-6

    Article  Google Scholar 

  24. Robertson, S.: The probabilistic relevance framework: BM25 and beyond. Found. Trends® Inf. Retr. 3(4), 333–389 (2010). https://doi.org/10.1561/1500000019

    Article  Google Scholar 

  25. Sackett, D.L.: Evidence-based medicine. Semin. Perinatol. 21(1), 3–5 (1997). https://doi.org/10.1016/S0146-0005(97)80013-4

    Article  Google Scholar 

  26. Scells, H., Zuccon, G., Deacon, A., Koopman, B.: QUT ielab at CLEF eHealth 2017 technology assisted reviews track: initial experiments with learning to rank. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  27. Singh, G., Marshall, I., Thomas, J., Wallace, B.: Identifying diagnostic test accuracy publications using a deep model. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  28. Sparck Jones, K., Sparck Jones, K., Walker, S., Walker, S., Robertson, S.E., Robertson, S.E.: A probabilistic model of information retrieval: development and comparative experiments part 2. Inf. Process. Manage. 36, 809–840 (2000). https://doi.org/10.1016/S0306-4573(00)00016-9

    Article  Google Scholar 

  29. Van Altena, A.J., Olabarriaga, S.D.: Predicting publication inclusion for diagnostic accuracy test reviews using random forests and topic modelling. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

  30. Wallace, B.C., Small, K., Brodley, C.E., Trikalinos, T.A.: Active learning for biomedical citation screening. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2010, p. 173 (2010). https://doi.org/10.1145/1835804.1835829

  31. Wallace, B.C., Trikalinos, T.A., Lau, J., Brodley, C.E., Schmid, C.H.: Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinform. 11(1), 55 (2010). https://doi.org/10.1186/1471-2105-11-55

    Article  Google Scholar 

  32. Yu, Z., Menzies, T.: Data balancing for technologically assisted reviews: undersampling or reweighting. In: CEUR Workshop Proceedings, vol. 1866 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Aristotle University of Thessaloniki, 54124, Thessaloniki, Greece

    Athanasios Lagopoulos, Antonios Anagnostou, Adamantios Minas & Grigorios Tsoumakas

Authors
  1. Athanasios Lagopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonios Anagnostou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adamantios Minas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Grigorios Tsoumakas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Athanasios Lagopoulos .

Editor information

Editors and Affiliations

  1. Aix-Marseille University, Marseille Cedex 20, France

    Patrice Bellot

  2. Virtual University of Tunis, Tunis, Tunisia

    Chiraz Trabelsi

  3. Systèmes d’informations, Big Data et Rec, Institut de Recherche en Informatique de, Toulouse Cedex 04, France

    Josiane Mothe

  4. Department of Computer Science, University of Huddersfield, Huddersfield, United Kingdom

    Fionn Murtagh

  5. DIRO, Universite de Montreal, Montreal, Québec, Canada

    Jian Yun Nie

  6. Pierre and Marie Curie University, Paris Cedex 05, France

    Laure Soulier

  7. Université d'Avignon et des Pays de, Avignon, France

    Eric SanJuan

  8. Department of Information Engineering, University of Padua, Padua, Padova, Italy

    Linda Cappellato

  9. University of Padua, Padua, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lagopoulos, A., Anagnostou, A., Minas, A., Tsoumakas, G. (2018). Learning-to-Rank and Relevance Feedback for Literature Appraisal in Empirical Medicine. In: Bellot, P., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2018. Lecture Notes in Computer Science(), vol 11018. Springer, Cham. https://doi.org/10.1007/978-3-319-98932-7_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-98932-7_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-98931-0

  • Online ISBN: 978-3-319-98932-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation