A Study of an Automatic Stopping Strategy for Technologically Assisted Medical Reviews
Systematic medical reviews are a method to collect the findings from multiple studies in a reliable way. Given budget and time constraints, limiting the recall of a search may undermine the quality of a review to such a degree that the validity of its findings is questionable. In this paper, we investigate a variable threshold approach to tackle the problem of a total recall task in medical reviews proposed by a Cross-Language Evaluation Forum (CLEF) eHealth lab in 2017. Compared to the official results submitted to the CLEF eHealth task, our approach performed consistently better over all the range of thresholds considered achieving a recall greater than 0.95 with 25,000 documents less than the best performing systems. The runs and the source code to generate the analyses of this paper are available at the following GitHub repository (https://github.com/gmdn/ECIR2018).
- 4.Grossman, M.R., Cormack, G.V., Roegiest, A.: TREC 2016 total recall track overview. In: Proceedings of the Twenty-Fifth TREC 2016, Gaithersburg, Maryland, USA, 15–18 November (2016)Google Scholar
- 5.Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2017 technologically assisted reviews in empirical medicine overview. In: Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, Dublin, Ireland, 11–14 September 2017. CEUR-WS.org (2017)
- 6.Di Nunzio, G.M., Beghini, F., Vezzani, F., Henrot, G.: An interactive two-dimensional approach to query aspects rewriting in systematic reviews. IMS Unipd At CLEF eHealth Task 2. In: Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, Dublin, Ireland, 11–14 September (2017)Google Scholar
- 9.Roegiest, A., Cormack, G.V., Grossman, M.R., Clarke, C.L.A.: TREC 2015 total recall track overview. In: Proceedings of the Twenty-fourth TREC 2015, Gaithersburg, Maryland, USA, 17–20 November (2015)Google Scholar