A Study of an Automatic Stopping Strategy for Technologically Assisted Medical Reviews

  • Giorgio Maria Di Nunzio
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10772)


Systematic medical reviews are a method to collect the findings from multiple studies in a reliable way. Given budget and time constraints, limiting the recall of a search may undermine the quality of a review to such a degree that the validity of its findings is questionable. In this paper, we investigate a variable threshold approach to tackle the problem of a total recall task in medical reviews proposed by a Cross-Language Evaluation Forum (CLEF) eHealth lab in 2017. Compared to the official results submitted to the CLEF eHealth task, our approach performed consistently better over all the range of thresholds considered achieving a recall greater than 0.95 with 25,000 documents less than the best performing systems. The runs and the source code to generate the analyses of this paper are available at the following GitHub repository (


  1. 1.
    Di Nunzio, G.M.: A new decision to take for cost-sensitive Naïve Bayes classifiers. Inf. Process. Manage. 50(5), 653–674 (2014)CrossRefGoogle Scholar
  2. 2.
    Di Nunzio, G.M.: Interactive text categorisation: the geometry of likelihood spaces. In: Lai, C., Giuliani, A., Semeraro, G. (eds.) Information Filtering and Retrieval. SCI, vol. 668, pp. 13–34. Springer, Cham (2017). CrossRefGoogle Scholar
  3. 3.
    Goeuriot, L., et al.: CLEF 2017 eHealth evaluation lab overview. In: Jones, G.J.F., et al. (eds.) CLEF 2017. LNCS, vol. 10456, pp. 291–303. Springer, Cham (2017). CrossRefGoogle Scholar
  4. 4.
    Grossman, M.R., Cormack, G.V., Roegiest, A.: TREC 2016 total recall track overview. In: Proceedings of the Twenty-Fifth TREC 2016, Gaithersburg, Maryland, USA, 15–18 November (2016)Google Scholar
  5. 5.
    Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2017 technologically assisted reviews in empirical medicine overview. In: Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, Dublin, Ireland, 11–14 September 2017. (2017)
  6. 6.
    Di Nunzio, G.M., Beghini, F., Vezzani, F., Henrot, G.: An interactive two-dimensional approach to query aspects rewriting in systematic reviews. IMS Unipd At CLEF eHealth Task 2. In: Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, Dublin, Ireland, 11–14 September (2017)Google Scholar
  7. 7.
    O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., Ananiadou, S.: Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst. Rev. 4(1), 5 (2015)CrossRefGoogle Scholar
  8. 8.
    Robertson, S.E., Zaragoza, H.: The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3(4), 333–389 (2009)CrossRefGoogle Scholar
  9. 9.
    Roegiest, A., Cormack, G.V., Grossman, M.R., Clarke, C.L.A.: TREC 2015 total recall track overview. In: Proceedings of the Twenty-fourth TREC 2015, Gaithersburg, Maryland, USA, 17–20 November (2015)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Information EngineeringUniversity of PaduaPaduaItaly

Personalised recommendations