Abstract
A Secondary Study (SS) is an important research method used in several areas. A crucial step in the Conduction phase of a SS is the search of studies. This step is time-consuming and error-prone, mainly due to the refinement of the search string. The objective of this study is to validate the effectiveness of an automatic formulation of search strings for SS. Our approach, termed Search String Generator (SeSG), takes as input a small set of studies (as a Quasi-Gold Standard) and processes them using text mining. After that, SeSG generates search strings that deliver a high F1-Score on the start set of a hybrid search strategy. To achieve this objective, we (1) generate a structured textual representation of the initial set of input studies as a bag-of-words using Term Frequency and Document Frequency; (2) perform automatic topic modeling using LDA (Latent Dirichlet Allocation) and enrichment of terms with a pre-trained dense language representation (embedding) called BERT (Bidirectional Encoder Representations from Transformers); (3) formulate and evaluate the search string using the obtained terms; and (4) use the developed search strings in a digital library. For the validation of our approach, we conduct an experiment—using some SS as objects—comparing the effectiveness of automatically formulated search strings by SeSG with manual search strings reported in these studies. SeSG generates search strings that achieve a better final F1-Score on the start set than the searches reported by these SS. Our study shows that SeSG can effectively supersede the formulation of search strings, in hybrid search strategies, since it dismisses the manual string refinements.
Similar content being viewed by others
References
Aggarwal CC, Zhai CX (2012) Mining text data. Springer Science+Business Media
Agrawal R, Srikant R (1994) Fast algorithms for mining association rules. In: Proceedings of the 20th international conference on very large data bases (VLDB), vol 1215, pp 487–499
Ali NB, Usman M (2018) Reliability of search in systematic reviews: towards a quality assessment framework for the automated-search strategy. Inf Softw Technol 99:133–147
Ampatzoglou A, Bibi S, Avgeriou P, Verbeek M, Chatzigeorgiou A (2019) Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Inf Softw Technol 106:201–230
Arampatzis A, Van Der Weide TP, van Bommel P, Koster CH (1999) Linguistically-motivated information retrieval. Encycl Library Inf Sci 69:201–222
Azeem MI, Palomba F, Shi L, Wang Q (2019) Machine learning techniques for code smell detection: a systematic literature review and meta-analysis. Inf Softw Technol 108:115–138
Babar MA, Zhang H (2009) Systematic literature reviews in software engineering: preliminary results from interviews with researchers. In: Proceedings of the 2009 3rd international symposium on empirical software engineering and measurement. IEEE Computer Society, pp 346–355
Badampudi D, Wohlin C, Petersen K (2015) Experiences from using snowballing and database searches in systematic literature studies. In: Proceedings of the 19th international conference on evaluation and assessment in software engineering, pp 1–10
Bengio Y, Ducharme R, Vincent P, Jauvin C (2003) A neural probabilistic language model. J Mach Learn Res 3:1137–1155
Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3:993–1022
Booth A (2016) Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev 5(1):74
Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH (2017) Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev 6(1):1–12
Briscoe S, Bethel A, Rogers M (2020) Conduct and reporting of citation searching in cochrane systematic reviews: a cross-sectional study. Res Synthesis Methods 11(2):169–180
Campbell DT, Cook TD (1979) Quasi-experimentation: design and analysis issues for field settings. Houghton Mifflin Company, Dallas
Chang AA, Heskett KM, Davidson TM (2006) Searching the literature using medical subject headings versus text word with pubmed. Laryngoscope 116(2):336–340
Collobert R, Weston J (2008) A unified architecture for natural language processing: deep neural networks with multitask learning. In: Proceedings of the 25th international conference on machine learning, pp 160–167
Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa P (2011) Natural language processing (almost) from scratch. J Mach Learn Res 12:2493–2537
Cooper C, Booth A, Britten N, Garside R (2017) A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev 6(1):1–16
Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R (2018) Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol 18(1):85
Devlin J, Chang MW, Lee K, Toutanova K (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: human language technologies, pp 4171–4186
Dickersin K, Scherer R, Lefebvre C (1994) Identifying relevant studies for systematic reviews. BMJ (Clinical research ed) 309:1286–91
Dieste O, Padua AG (2007) Developing search strategies for detecting relevant experiments for systematic reviews. In: First international symposium on empirical software engineering and measurement (ESEM 2007), pp 215–224
Dieste O, Grimán A, Juristo N (2009) Developing search strategies for detecting relevant experiments. Empir Softw Eng 14(5):513–539. https://doi.org/10.1007/s10664-008-9091-7
Feng L, Chiam YK, Lo SK (2017) Text-mining techniques and tools for systematic literature reviews: a systematic literature review. In: 2017 24th Asia-Pacific software engineering conference (APSEC), pp 41–50
Ghafari M, Saleh M, Ebrahimi T (2012) A federated search approach to facilitate systematic literature review in software engineering. Int J Softw Eng Appl (IJSEA) 3(2):13–24
Gonzalez MAI, de Lima VLS, de Lima JV (2006) Tools for nominalization: an alternative for lexical normalization. In: Computational processing of the Portuguese language, pp 100–109
Grames EM, Stillman AN, Tingley MW, Elphick CS (2019) An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks. Methods Ecol Evol 10(10):1645–1654
Haynes RB, Kastner M, Wilczynski NL (2005) Developing optimal search strategies for detecting clinically sound and relevant causation studies in embase. BMC Med Inform Decis Making 5(1):1–7
Horsley T, Dingwall O, Sampson M (2011) Checking reference lists to find additional studies for systematic reviews. Cochrane Database of Systematic Reviews (8)
Hosseini S, Turhan B, Gunarathna D (2019) A systematic literature review and meta-analysis on cross project defect prediction. IEEE Trans Softw Eng 45(2):111–147
Imtiaz S, Bano M, Ikram N, Niazi M (2013) A tertiary study: experiences of conducting systematic literature reviews in software engineering. In: Proceedings of the 17th international conference on evaluation and assessment in software engineering, pp 177–182
Jones KS, Willett P (1997) Readings in information retrieval. Morgan Kaufmann Publishers Inc., San Francisco
Juristo N, Moreno AM (2001) Basics of software engineering experimentation. Springer Science & Business Media
Kitchenham B (2004) Procedures for performing systematic reviews, vol 33. Keele University, Keele, pp 1–26
Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report—Department of Computer Science, University of Durham
Kitchenham BA, Li Z, Burn AJ (2011) Validating search processes in systematic literature reviews. In: EAST, pp 3–9
Kitchenham BA, Budgen D, Brereton P (2015) Evidence-based software engineering and systematic reviews. Chapman & Hall/CRC
Krüger J, Lausberger C, von Nostitz-Wallwitz I, Saake G, Leich T (2020) Search. Review. Repeat? An empirical study of threats to replicating slr searches. Empir Softw Eng 25(1):627–677
Krovetz R (1993) Viewing morphology as an inference process. In: Proceedings of the 16th annual international ACM SIGIR conference on research and development in information retrieval, pp 191–202
Kuhrmann M, Fernández DM, Daneva M (2017) On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empir Softw Eng 22(6):2852–2891
Kuper H, Nicholson A, Hemingway H (2006) Searching for observational studies: what does citation tracking add to pubmed? A case study in depression and coronary heart disease. BMC Med Res Methodol 6(1):1–4
Laguna MdSC, Pardo TAS, Rezende SO (2014) Extração automática de termos simples baseada em aprendizado de máquina. Doctoral thesis in ciências de computação e matemática computacional, Instituto de Ciências Matemáticas e de Computação, University of São Paulo, São Carlos, SP
Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, Kang J (2019) Biobert: a pre-trained biomedical language representation model for biomedical text mining. arXiv:190108746
Luhn HP (1958) The automatic creation of literature abstracts. IBM J Res Dev 2(2):159–165
Manning C, Raghavan P, Schütze H (2010) Introduction to information retrieval. Nat Lang Eng 16(1):100–103
Marcos-Pablos S, García-Peñalvo FJ (2018) Information retrieval methodology for aiding scientific database search. Soft Comput 1–10
Marshall C, Brereton P (2013) Tools to support systematic literature reviews in software engineering: a mapping study. In: 2013 ACM/IEEE international symposium on empirical software engineering and measurement, pp 296–299
Marshall C, Brereton P, Kitchenham B (2014) Tools to support systematic reviews in software engineering: a feature analysis. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, association for computing machinery
McCann B, Bradbury J, Xiong C, Socher R (2017) Learned in translation: contextualized word vectors. In: Proceedings of the 31st international conference on neural information processing systems, pp 6297–6308
Mergel GD, Silveira MS, da Silva TS (2015) A method to support search string building in systematic literature reviews through visual text mining. In: Proceedings of the 30th annual ACM symposium on applied computing. Association for Computing Machinery, pp 1594–1601
Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Proceedings of the 26th international conference on neural information processing systems, vol 2, pp 3111–3119
Mikolov T, Grave E, Bojanowski P, Puhrsch C, Joulin A (2018) Advances in pre-training distributed word representations. In: Proceedings of the eleventh international conference on language resources and evaluation. European Language Resources Association (ELRA)
Montgomery DC (2017) Design and analysis of experiments. Wiley, New York
Montgomery DC, Runger GC (2018) Applied statistics and probability for engineers. Wiley, New York
Mourão E, Kalinowski M, Murta L, Mendes E, Wohlin C (2017) Investigating the use of a hybrid search strategy for systematic reviews. In: 2017 ACM/IEEE international symposium on empirical software engineering and measurement, pp 193–198
Mourão E, Pimentel JF, Murta L, Kalinowski M, Mendes E, Wohlin C (2020) On the performance of hybrid search strategies for systematic literature reviews in software engineering. Inf Softw Technol 106–294
Münch J, Armbrust O, Kowalczyk M, Soto M (2012) Software process definition and management. Springer Publishing Company, Incorporated
Nogueira BM (2009) Avaliação de métodos não-supervisionados de seleção de atributos para mineração de textos. Doctoral thesis in ciências de computação e matemática computacional, Instituto de ciências matemáticas e de computação University of São Paulo, São Carlos
Nogueira BM (2013) Hierarchical semi-supervised confidence-based active clustering and its application to the extraction of topic hierarchies from document collections. Masters dissertation in ciências de computação e matemática computacional, Instituto de ciências matemáticas e de computação University of São Paulo São Carlos
O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S (2015) Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev 4(1):5
Pennington J, Socher R, Manning C (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543
Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. In: Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 1, pp 2227–2237
Petitti DB, et al. (2000) Meta-analysis, decision analysis, and cost-effectiveness analysis: methods for quantitative synthesis in medicine. 31, OUP USA
Pravin A, Srinivasan S (2012) Detecting of software bugs in source code using data mining approach. Natl J Syst Inf Technol 6(1):1–8
Relevo R (2012) Effective search strategies for systematic reviews of medical tests. J Gen Internal Med 27(1):28–32
Rezende SO (2003) Sistemas inteligentes: fundamentos e aplicações. Editora Manole Ltda, Barueri
Rogers A, Kovaleva O, Rumshisky A (2020) A primer in bertology: what we know about how bert works. Trans Assoc Comput Linguist 8:842–866. https://doi.org/10.1162/tacl_a_00349
Ros R, Bjarnason E, Runeson P (2017) A machine learning approach for semi-automated search and selection in literature studies. In: Proceedings of the 21st international conference on evaluation and assessment in software engineering. Association for Computing Machinery, pp 118–127
Salton G, Wong A, Yang CS (1975) A vector space model for automatic indexing. Commun ACM 18(11):613–620
Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591–611
Shearer C (2000) The crisp-dm model: the new blueprint for data mining. J Data Warehous 5(4):13–22
Smalheiser NR, Lin C, Jia L, Jiang Y, Cohen AM, Yu C, Davis JM, Adams CE, McDonagh MS, Meng W (2014) Design and implementation of metta, a metasearch engine for biomedical literature retrieval intended for systematic reviewers. Health Inf Sci Syst 2(1):1
Stansfield C, O’Mara-Eves A, Thomas J (2017) Text mining for search term development in systematic reviewing: a discussion of some methods and challenges. Res Synth Methods 8(3):355–365
Sullivan GM, Feinn R (2012) Using effect size—or why the p value is not enough. J Grad Med Educ 4(3):279–282
Tomassetti F, Rizzo G, Vetro A, Ardito L, Torchiano M, Morisio M (2011) Linked data approach for selection process automation in systematic reviews. In: 15th Annual conference on evaluation & assessment in software engineering (EASE 2011), pp 31–35
Trochim WM, Donnelly JP (2020) Research methods knowledge base. https://conjointly.com/kb/ (version current as of 27 April 2020)
Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E (2014) Systematic review automation technologies. Syst Rev 3(1):74
Turian J, Ratinov L, Bengio Y (2010) Word representations: a simple and general method for semi-supervised learning. In: Proceedings of the 48th annual meeting of the association for computational linguistics. Association for Computational Linguistics, pp 384–394
van Rijsbergen C (1979) Information retrieval. http://www.dcs.gla.ac.uk/Keith/Preface.html. Accessed 7 July 2020
Vasconcellos FJ, Landre GB, Cunha JAO, Oliveira JL, Ferreira RA, Vincenzi AM (2017) Approaches to strategic alignment of software process improvement: a systematic literature review. J Syst Softw 123:45–63
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser U, Polosukhin I (2017) Attention is all you need. In: Proceedings of the 31st international conference on neural information processing systems, , NIPS’17. Curran Associates Inc., Red Hook, pp 6000–6010
Wang J, Wang Q (2016) Analyzing and predicting software integration bugs using network analysis on requirements dependency network. Requir Eng 21(2):161–184
Weiss SM, Indurkhya N (1998) Predictive data mining: a practical guide. Morgan Kaufmann Publishers Inc., San Francisco
Weiss SM, Indurkhya N, Zhang T, Damerau F (2010) Text mining: predictive methods for analyzing unstructured information, 1st edn. Springer Publishing Company, Incorporated
White VJ, Glanville JM, Lefebvre C, Sheldon TA (2001) A statistical approach to designing search filters to find systematic reviews: objectivity enhances accuracy. J Inf Sci 27(6):357–370
Wieringa RJ (2014) Design science methodology for information systems and software engineering. Springer, Berlin
Wieringa R, Daneva M (2015) Six strategies for generalizing software engineering theories. Sci Comput Program 101:136–152
Wilczynski NL, Haynes RB (2005) Embase search strategies for identifying methodologically sound diagnostic studies for use by clinicians and researchers. BMC Med 3(1):1–6
Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering. Association for Computing Machinery
Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Publishing Company, Incorporated
Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Davison J, Shleifer S, von Platen P, Ma C, Jernite Y, Plu J, Xu C, Le Scao T, Gugger S, Drame M, Lhoest Q, Rush A (2020) Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. Association for Computational Linguistics, pp 38–45
Yang L, Zhang H, Shen H, Huang X, Zhou X, Rong G, Shao D (2021) Quality assessment in systematic literature reviews: a software engineering perspective. Inf Softw Technol 130:106397. https://doi.org/10.1016/j.infsof.2020.106397. https://www.sciencedirect.com/science/article/pii/S0950584920301610
Zhang H, Babar MA (2011) An empirical investigation of systematic reviews in software engineering. In: 2011 International symposium on empirical software engineering and measurement. https://doi.org/10.1109/ESEM.2011.17, pp 87–96
Zhang H, Babar MA, Tell P (2011) Identifying relevant studies in software engineering. Inf Softw Technol 53(6):625–637
Zwakman M, Verberne LM, Kars MC, Hooft L, van Delden JJ, Spijker R (2018) Introducing palette: an iterative method for conducting a literature search for a review in palliative care. BMC Palliative Care 17(1):1–9
Acknowledgements
The authors would like to thank the Coordination for the Improvement of Higher Education Personnel (CAPES) for financially assisting this study.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Tim Menzies
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This work was supported by the Coordination for the Improvement of Higher Education Personnel (CAPES) Grant No. 88882.466597/2019-01.
Rights and permissions
About this article
Cite this article
Alves, L.F., Vasconcellos, F.J.S. & Nogueira, B.M. SeSG: a search string generator for Secondary Studies with hybrid search strategies using text mining. Empir Software Eng 27, 105 (2022). https://doi.org/10.1007/s10664-021-10084-4
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-021-10084-4