Abstract
High recall Information REtrieval (HIRE) aims at identifying only and (almost) all relevant documents for a given query. HIRE is paramount in applications such as systematic literature review, medicine, legal jurisprudence, among others. To address the HIRE goals, active learning methods have proven valuable in determining informative and non-redundant documents to reduce user effort for manual labeling. We propose a new active learning framework for the HIRE task. REVEAL-HIRE selects a very reduced set of documents to be labeled, significantly mitigating the user’s effort. The proposed approach selects the most representative documents by exploiting a novel, specifically designed active learning strategy for HIRE, called REVEAL (RelEVant rulE-based Active Learning). REVEAL aims at selecting the maximum number of relevant documents for a given query based on discriminative rule-based patterns and a penalization factor. The method is applied to the top-ranked documents to choose the most informative ones to be labeled, a hard task due to data skewness – most documents are irrelevant for a given query. The enhanced active learning process is repeated incrementally until a stopping point is achieved, using REVEAL to identify the point in the process when relevant documents should stop to be sampled. Experimental results in several standard benchmark datasets (e.g. 20-Newsgroups, Trec Total Recall, and CLEF eHealth) demonstrate that REVEAL-HIRE can reduce the user labeling effort up to 3 times (320% of reduction) in comparison with state-of-the-art baselines while keeping the effectiveness at the highest levels.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Electronic discovery is the process of obtaining and exchanging evidence in a legal case or investigation.
References
Cohen, A.M., Hersh, W.R., Peterson, K., & et al. (2006). Reducing workload in systematic review preparation using automated citation classification. Journal of the American Medical Informatics Association, 13(2), 206–219. https://doi.org/10.1197/jamia.M1929.
Cormack, G.V., & Grossman, M.R. (2014). Evaluation of machine-learning protocols for technology-assisted review in electronic discovery. In Proceedings of the 37th international ACM SIGIR conference on research development in information retrieval. SIGIR ’14 (pp. 153–162). New York, ACM. https://doi.org/10.1145/2600428.2609601.
Cormack, G.V., & Grossman, M.R. (2016a). Scalability of continuous active learning for reliable high-recall text classification. In Proceedings of the 25th ACM international on conference on information and knowledge management. CIKM ’16 (pp. 1039–1048). New York, Association for Computing Machinery. https://doi.org/10.1145/2983323.2983776.
Cormack, G.V., & Grossman, M.R. (2016b). Engineering quality and reliability in technology-assisted review. In Proceedings of the 39th international ACM SIGIR conference on research and development in information retrieval. SIGIR ’16 (pp. 75–84). https://doi.org/10.1145/2911451.2911510.
Dal Bianco, G., Galante, R., Goncalves, M.A., & et al. (2015). A practical and effective sampling selection strategy for large scale deduplication. IEEE Transactions on Knowledge and Data Engineering, 27(9), 2305–2319. https://doi.org/10.1109/ICDE.2016.7498403.
Devi, D., Biswas, S.K., & Purkayastha, B. (2020). A review on solution to class imbalance problem: undersampling approaches. In International conference on computational performance evaluation (ComPE) (pp. 626–631). New York, ACM. https://doi.org/10.1145/2983323.2983776.
Di Nunzio, G.M., & Faggioli, G. (2021). A study of a gain based approach for query aspects in recall oriented tasks. Applied Sciences, 11(19), 9075. https://doi.org/10.3390/app11199075.
Dimitrakis, E., Sgontzos, K., & Tzitzikas, Y. (2020). A survey on question answering systems over linked data and documents. Journal of Intelligent Information Systems, 55(2), 233–259. https://doi.org/10.1007/s10844-019-00584-7.
Dinter, V., Tekinerdogan, B., & Catal, C. (2021). Automation of systematic literature reviews: a systematic literature review. Information and Software Technology, 136, 1071–1081. https://doi.org/10.1016/j.infsof.2021.106589.
Fang, M., & Zhu, X. (2014). Active learning with uncertain labeling knowledge. Pattern Recognition Letters, 43, 98–108. https://doi.org/10.1016/j.patrec.2013.10.011.
Felizardo, K.R., Mendes, E., Kalinowski, M., & et al. (2016). Using forward snowballing to update systematic reviews in software engineering. In Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement. ESEM ’16. New York, ACM. https://doi.org/10.1145/2961111.2962630.
Fu, Y., Zhu, X., & Li, B. (2013). A survey on instance selection for active learning. Knowledge and Information Systems, 35(2), 249–283. https://doi.org/10.1007/s10115-012-0507-8.
Hanna, S., Liadh, K., Lorraine, G., & et al. (2018). Overview of the clef ehealth evaluation lab 2018. In International conference of the cross-language evaluation forum for european languages (pp. 286–301). Cham, Springer. https://doi.org/10.1007/978-3-319-98932-7_26.
Haotian, Z., Wu, L., Yipeng, W., & et al. (2015). Waterlooclarke: trec 2015 total recall trackwaterlooclarke: trec 201. In TREC.
Hassler, E.E., Hale, D.P., & Hale, J.E. (2018). A comparison of automated training-by-example selection algorithms for evidence based software engineering. Information and Software Technology, 98, 59–73. https://doi.org/10.1016/j.infsof.2018.02.001.
Iovine, A., Lops, P., Narducci, F., & et al. (2022). An empirical evaluation of active learning strategies for profile elicitation in a conversational recommender system. Journal of Intelligent Information System, 58(2), 337–362. https://doi.org/10.1007/s10844-021-00683-4.
Kontonatsios, G., Spencer, S., Matthew, P., & Korkontzelos, I. (2020). Using a neural network-based feature extraction method to facilitate citation screening for systematic reviews. Expert Systems with Applications, 6, 100030. https://doi.org/10.1016/j.eswax.2020.100030.
Lewis, D.D., Yang, E., & Frieder, O. (2021). Certifying one-phase technology-assisted reviews. In Proceedings of the 30th ACM international conference on information & knowledge management (pp. 893–902). New York, ACM. https://doi.org/10.1145/3459637.3482415.
Li, D., & Kanoulas, E. (2020). When to stop reviewing in technology-assisted reviews: sampling from an adaptive distribution to estimate residual relevant documents. ACM Transactions on Information Systems (4). https://doi.org/10.1145/3411755.
Li, P., Zhang, Y., & Zhang, B. (2022). Understanding query combination behavior in exploratory searches. Applied Sciences, 12 (2), 706. https://doi.org/10.3390/app12020706.
Manning, C., Raghavan, P., & Schütze, H. (2010). Introduction to information retrieval. Natural Language Engineering, 16(1), 100–103.
Mourão, E., Pimentel, J., Murta, L., & et al. (2020). On the performance of hybrid search strategies for systematic literature reviews in software engineering. Information and Software Technology, 123, 106294. https://doi.org/10.1016/j.infsof.2020.106294.
Nunzio, G.M.D. (2020). A study on a stopping strategy for systematic reviews based on a distributed effort approach. In International conference of the cross-language evaluation forum for european languages (pp. 112–123). https://doi.org/10.1007/978-3-030-58219-7_10.
Olorisade, B.K., Brereton, P., & Andras, P. (2019). The use of bibliography enriched features for automatic citation screening. Journal of Biomedical Informatics, 94, 103202. https://doi.org/10.1016/j.jbi.2019.103202.
Roegiest, A., Cormack, G.V., Clarke, C.L., & Grossman, M.R. (2015). Trec 2015 total recall track overview. In TREC.
Silva, R.M., Gomes, G.C., Alvim, M.S., & et al. (2016). Compression-based selective sampling for learning to rank. In Proceedings of the 25th ACM international conference on information and knowledge management (pp. 247–256). New York, ACM. https://doi.org/10.1145/2983323.2983813.
Silva, R.M., Gomes, G.C.M., Alvim, M.S., & et al. (2022). How to build high quality L2R training data: unsupervised compression-based selective sampling for learning to rank. Information Sciences, 601, 90–113. https://doi.org/10.1016/j.ins.2022.04.012.
Silva, R.M., Gonçalves, M.A., & Veloso, A. (2014). A two-stage active learning method for learning to rank. Journal of the Association for Information Science and Technology, 65(1), 109–128. https://doi.org/10.1002/asi.22958.
Song, J.J., Lee, W., & Afshar, J. (2019). An effective high recall retrieval method. Data Knowledge Engineering, 101603. https://doi.org/10.1016/j.datak.2017.07.006.
Tim, W., Till, S., Dennis, O., & et al. (2020). A clustering approach for topic filtering within systematic literature reviews. MethodsX, 7, 100831. https://doi.org/10.1016/j.mex.2020.100831.
Trotman, A. (2005). Learning to rank. Information Retrieval, 8(3), 359–381. https://doi.org/10.1007/s10791-005-6991-7.
van Dinter, R., Catal, C., & Tekinerdogan, B. (2021). A multi-channel convolutional neural network approach to automate the citation screening process. Applied Soft Computing, 112, 107765. https://doi.org/10.1016/j.asoc.2021.107765.
Wallace, B.C., Small, K., Brodley, C.E., & et al. (2010). Active learning for biomedical citation screening. In Proceedings of the 16th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 173–182). New York, ACM. https://doi.org/10.1145/1835804.1835829.
Wei, K., Iyer, R., & Bilmes, J. (2015). Submodularity in data subset selection and active learning. In Proceedings of the 32nd international conference on international conference on machine learning. ICML’15, (vol. 37 pp. 1954–1963). New York, JMLR.org. https://doi.org/10.5555/3045118.3045326.
Yang, E., Lewis, D.D., & Frieder, O (2021). Heuristic stopping rules for technology-assisted review. In Proceedings of the 21st ACM Symposium on Document Engineering. DocEng ’21. Limerick, ACM. https://doi.org/10.1145/3469096.3469873.
Yu, Z., & Menzies, T. (2019). Fast2: an intelligent assistant for finding relevant papers. Expert Systems with Applications, 120, 57–71. https://doi.org/10.1016/j.eswa.2018.11.021.
Zhang, H., Abualsaud, M., Ghelani, N., & et al. (2018). Effective user interaction for high-recall retrieval: less is more. In Proceedings of the 27th ACM international conference on information and knowledge management. CIKM ’18 (pp. 187–196). New York, ACM. https://doi.org/10.1145/3269206.3271796.
Zhang, H., Cormack, G.V., Grossman, M.R., & et al. (2020). Evaluating sentence-level relevance feedback for high-recall information retrieval. Information Retrieval Journal, 23(1), 1–26. https://doi.org/10.1007/s10791-019-09361-0.
Zuobing, X., Ram, A., & Yi, Z. (2007). Incorporating diversity and density in active learning for relevance feedback. In European conference on information retrieval (pp. 246–257). Berlin, Springer. https://doi.org/10.1007/978-3-540-71496-5_24.
Acknowledgements
This work was partially supported by CAPES, CNPq, FAPEMIG, NVIDIA and Google.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Marcos André Gonçalves and Denio Duarte contributed equally to this work.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Bianco, G.D., Duarte, D. & Gonçalves, M.A. Reducing the user labeling effort in effective high recall tasks by fine-tuning active learning. J Intell Inf Syst 61, 453–472 (2023). https://doi.org/10.1007/s10844-022-00772-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10844-022-00772-y