Skip to main content
Log in

Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable?

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript


The systematic review (SR) is a methodology used to find and aggregate all relevant existing evidence about a specific research question of interest. One of the activities associated with the SR process is the selection of primary studies, which is a time consuming manual task. The quality of primary study selection impacts the overall quality of SR. The goal of this paper is to propose a strategy named “Score Citation Automatic Selection” (SCAS), to automate part of the primary study selection activity. The SCAS strategy combines two different features, content and citation relationships between the studies, to make the selection activity as automated as possible. Aiming to evaluate the feasibility of our strategy, we conducted an exploratory case study to compare the accuracy of selecting primary studies manually and using the SCAS strategy. The case study shows that for three SRs published in the literature and previously conducted in a manual implementation, the average effort reduction was 58.2 % when applying the SCAS strategy to automate part of the initial selection of primary studies, and the percentage error was 12.98 %. Our case study provided confidence in our strategy, and suggested that it can reduce the effort required to select the primary studies without adversely affecting the overall results of SR.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others


  1. Instructions for using Weka and applying SCAS in StArt are available at


  • Babar MA, Zhang H (2010) Systematic literature reviews in software engineering: Preliminary results from interviews with researchers. In: Proc, ESEM’09, 1–10

  • Boell S, Cezec-Kecmanovic D (2011) “Are systematic reviews better, less biased and of higher quality?” ECIS 2011 Proceedings. Paper 223.

  • Brereton OP, Kitchenham BA, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(1):571–583

    Article  Google Scholar 

  • Carletta J (1996) Assessing agreement on classification tasks: the kappa statistic. Comput Linguis 22(2):249–254

    Google Scholar 

  • Dieste O, Grimán A, Juristo N (2009) Developing search strategies for detecting relevant experiments. In: Proc. ESEM’09, 513–539

  • Dybå T, Kitchenham BA, Jorgensen M (2005) Evidence-based software engineering for practitioners. IEEE Softw 22(1):58–65

    Article  Google Scholar 

  • Dybå T, Dingsøyr T, Hanssen GK (2007) Applying systematic reviews to diverse study types: An experience report. In: Proc. ESEM’07, 225–234

  • Fabbri S, Hernandes E, Di Thommazo A, Belgamo A, Zamboni A, Silva C (2012) Managing literature reviews information through visualization. In: Proc. ICEIS’12, 36–45

  • Felizardo KR, Salleh N, Martins R, Mendes E, MacDonell S, Maldonado JC (2011) Using visual text mining to support the study selection activity in systematic literature reviews. In: Proc. ESEM’11, 1–10

  • Felizardo KR, Andery G, Paulovich F, Minghim R, Maldonado JC (2012) A visual analysis approach to validate the selection review of primary studies in systematic reviews. Inf Softw Technol 54(10):1079–1091

    Article  Google Scholar 

  • Kitchenham BA, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. EBSE Technical Report, Keele University and University of Durham, version 2.3

  • Kitchenham BA, Dybå T, Jørgensen M (2004) Evidence-based software engineering. In: Proc. ICSE’08, 273–281

  • Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174

    Article  MATH  MathSciNet  Google Scholar 

  • Malheiros V, Hohn E, Pinho R, Mendonca M, Maldonado JC (2007) A visual text mining approach for systematic reviews. In: Proc. ESEM’07, 245–254

  • Marshall C, Brereton P (2013) Tools to Support Systematic Literature Reviews in Software Engineering: A Mapping Study. In: Proc. ESEM’13, 296–299

  • Oates BJ, Capper G (2009) Using systematic reviews and evidence-based software engineering with masters students. In: Proc. EASE’09, 1–8

  • Paulovich FV, Minghim R (2008) HiPP: a novel hierarchical point placement strategy and its application to the exploration of document collections. IEEE Trans Vis Comput Graph 14(6):1229–1236

    Article  Google Scholar 

  • Petersen K, Ali NB (2011) Identifying strategies for study selection in systematic reviews and maps. In: Proc. ESEM’11, 351–354

  • Riaz M, Sulayman M, Salleh N, Mendes E (2010) Experiences conducting systematic reviews from novices’ perspective. In: Proc. EASE’10, 1–10

  • Shaw M, Clements P (2006) The golden age of software architecture: A comprehensive survey. Technical Report CMU-ISRI-06-101, Software Engineering Institute, Carnegie Mellon University

  • Shepperd M (2007) Software project economics: A roadmap. In: Proc. FOSE’07, IEEE Computer Society, Washington, DC, pp 304–315

  • Weka (2013) Available at Last access on 30-March-2013

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in Software Engineering: An Introduction. In: Kluwer Academic Publishers (ed), 1st edn. Boston, US, pp 63–74

  • Zhang H, Muhammad AB (2011) An empirical investigation of systematic reviews in software engineering. In: Proc. ESEM’11, 1–10

Download references


This research is supported by the Brazilian funding agency: FAPESP (Process no. 2012/02524-4).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Fábio R. Octaviano.

Additional information

Communicated by: Tony Gorschek

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Octaviano, F.R., Felizardo, K.R., Maldonado, J.C. et al. Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable?. Empir Software Eng 20, 1898–1917 (2015).

Download citation

  • Published:

  • Issue Date:

  • DOI: