Advertisement

Identification and prioritization of SLR search tool requirements: an SLR and a survey

  • Ahmed Al-Zubidy
  • Jeffrey C. Carver
Article

Abstract

Context

The number of published systematic literature reviews (SLRs) in software engineering venues is increasing. However, even with their high adaptation rate, the task of performing an SLR requires a large amount of effort and presents a number of barriers. Specifically, during the SLR search phase authors must expend a lot of time and overcome a large number of barriers.

Objective

To help alleviate some of the barriers in the search phase, we identify and prioritize SLR search tool requirements based on input from the SLR community. These requirements will help tool builders ensure they focus their efforts appropriately.

Method

We conducted an SLR and a survey of SLR authors in software engineering. In the SLR we extracted problems and solutions SLR authors reported during their search processes. In the survey we asked respondents to describe the problems they faced during SLR search and to specify any requirements they would like to see as part of an SLR search tool. We also asked survey respondents to describe any tools they use to support SLR search, along with the strengths and limitations of those tools.

Results

Based on analysis of 84 studies from the literature and a qualitative analysis of 131 responses from researchers, we identified a set of functional requirements that authors, researchers, and tool builders can use as a reference. We also describe the tools currently used by SLR researchers.

Conclusions

The list of SLR search tool requirements can be used by tool builders as a guide when constructing new tools. Our analysis of tools showed that more recent tools are covering more of the requirements than in the past.

Keywords

Systematic literature review Empirical software engineering Tooling infrastructure 

Notes

Acknowledgments

We thank the survey respondents for taking the time to complete the survey. We also acknowledge partial support from the United States National Science Foundation grant number 1305395.

References

  1. Al-Zubidy A, Carver JC, Hale DP, Hassler EE (2017) Vision for {SLR} tooling infrastructure: prioritizing value-added requirements. Inf Softw Technol.  https://doi.org/10.1016/j.infsof.2017.06.007
  2. Allman-Farinelli M, Byron A, Collins C, Gifford J, Williams P (2014) Challenges and lessons from systematic literature reviews for the australian dietary guidelines. Aust J Prim Health 20(3):236–240CrossRefGoogle Scholar
  3. Babar MA, Zhang H (2009) Systematic literature reviews in software engineering: preliminary results from interviews with researchers. In: 3rd international symposium on empirical software engineering and measurement. IEEE Computer Society, pp 346–355Google Scholar
  4. Badampudi D, Wohlin C, Petersen K (2015) Experiences from using snowballing and database searches in systematic literature studies. In: 19th international conference on evaluation and assessment in software engineering. ACM, p 17Google Scholar
  5. Barat S, Clark T, Barn B, Kulkarni V (2017) A model-based approach to systematic review of research literature. In: Proceedings of the 10th innovations in software engineering conference. ACM, pp 15–25Google Scholar
  6. Barn B, Raimondi F, Athappian L, Clark T (2014) Slrtool: a tool to support collaborative systematic literature reviews. In: ICEIS 2014—Proceedings of the 16th international conference on enterprise information systems, vol 2, pp 440–447Google Scholar
  7. Biolchini J, Mian PG, Natali ACC, Travassos GH (2005) Systematic review in software engineering. System Engineering and Computer Science Department COPPE/UFRJ. Technical Report ES 679(05):45Google Scholar
  8. Bowes D, Hall T, Beecham S (2012) Slurp: a tool to help large complex systematic literature reviews deliver valid and rigorous results. In: Proceedings of the 2nd international workshop on evidential assessment of software technologies. ACM, pp 33–36Google Scholar
  9. Brereton P, Kitchenham BA, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(4):571–583CrossRefGoogle Scholar
  10. Budgen D, Brereton P, Drummond S, Williams N (2018) Reporting systematic reviews: some lessons from a tertiary study. Inf Softw Technol 95:62–74.  https://doi.org/10.1016/j.infsof.2017.10.017 CrossRefGoogle Scholar
  11. Carver JC, Hassler E, Hernandes E, Kraft NA (2013) Identifying barriers to the systematic literature review process. In: 7th international symposium on empirical software engineering and measurement. IEEE, pp 203–212Google Scholar
  12. Corbin J, Strauss A et al (2008) Basics of qualitative research: techniques and procedures for developing grounded theory. Thousand OaksGoogle Scholar
  13. de Almeida Biolchini JC, Mian PG, Natali ACC, Conte TU, Travassos GH (2007) Scientific research ontology to support systematic review in software engineering. Adv Eng Inf 21(2):133–151CrossRefGoogle Scholar
  14. Dieste O, Grimán A, Juristo N (2009) Developing search strategies for detecting relevant experiments. Empir Softw Eng 14(5):513–539CrossRefGoogle Scholar
  15. Dyba T, Dingsoyr T, Hanssen GK (2007) Applying systematic reviews to diverse study types: an experience report. In: 1st international symposium on empirical software engineering and measurement. IEEE, pp 225–234Google Scholar
  16. Engström E, Skoglund M, Runeson P (2008) Empirical evaluations of regression test selection techniques: a systematic review. In: 1st international symposium on empirical software engineering and measurement. ACM, pp 22–31Google Scholar
  17. Felizardo KR, Mendes E, Kalinowski M, Souza ÉF, Vijaykumar NL (2016) Using forward snowballing to update systematic reviews in soft. engineering. In: Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement. ACM, p 53Google Scholar
  18. Fernández-Sáez A M, Bocco MG, Romero FP (2010) Slr-tool: a tool for performing systematic literature reviews. In: ICSOFT (2), pp 157–166Google Scholar
  19. Hassler E, Carver JC, Kraft NA, Hale D (2014) Outcomes of a community workshop to identify and rank barriers to the systematic literature review process. In: 18th International conference on evaluation & assessment in software engineering. ACM, p 31Google Scholar
  20. Hassler E, Carver JC, Hale D, Al-Zubidy A (2016) Identification of SLR tool needs–results of a community workshop. J Inf Softw Technol 70:122–129CrossRefGoogle Scholar
  21. Hernandes E, Zamboni A, Fabbri S, Thommazo AD (2012) Using gqm and tam to evaluate start-a tool that supports systematic review. CLEI Electron J 15 (1):3–3Google Scholar
  22. Jalali S, Wohlin C (2012) Systematic literature studies: database searches vs. backward snowballing. In: Proceedings of the ACM-IEEE international symposium on empirical software engineering and measurement. ACM, pp 29–38Google Scholar
  23. Kitchenham B (2004) Procedures for performing systematic reviews. Tech. Rep. TR/SE-0401, Keel UniversityGoogle Scholar
  24. Kitchenham B, Brereton P (2013) A systematic review of systematic review process research in software engineering. Inf Softw Technol 55(12):2049–2075CrossRefGoogle Scholar
  25. Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Tech. Rep. EBSE-2007-01, Keel UniversityGoogle Scholar
  26. Kitchenham B, Linkman S, Law D (1997) Desmet: a methodology for evaluating software engineering methods and tools. Comput Control Eng J 8(3):120–126CrossRefGoogle Scholar
  27. Kitchenham BA, Dyba T, Jorgensen M (2004) Evidence-based software engineering. In: 26th international conference on software engineering. IEEE Computer Society, pp 273–281Google Scholar
  28. Kitchenham B, Pretorius R, Budgen D, Brereton OP, Turner M, Niazi M, Linkman S (2010) Systematic literature reviews in software engineering–a tertiary study. Inf Softw Technol 52(8):792– 805CrossRefGoogle Scholar
  29. Kitchenham BA, Budgen D, Brereton P (2015) Evidence-based software engineering and systematic reviews, vol 4. CRC Press, Boca RatonGoogle Scholar
  30. Mallett R, Hagen-Zanker J, Slater R, Duvendack M (2012) The benefits and challenges of using systematic reviews in international development research. J Dev Effect 4(3):445–455CrossRefGoogle Scholar
  31. Marshall C, Brereton P (2013) Tools to support systematic literature reviews in software engineering: a mapping study. In: 7th international symposium on empirical software engineering and measurement. IEEE, pp 296–299Google Scholar
  32. Marshall C, Brereton P, Kitchenham B (2014) Tools to support systematic reviews in software engineering: a feature analysis. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering. ACM, p 13Google Scholar
  33. Molléri JS, Benitti FBV (2015) Sesra: a web-based automated tool to support the systematic literature review process. In: Proceedings of the 19th international conference on evaluation and assessment in software engineering. ACM, p 24Google Scholar
  34. Parsifal (2015) Parsifal systematic literature review tool. [Online; accessed 1–December-2016]Google Scholar
  35. Perestelo-Pérez L (2013) Standards on how to develop and report systematic reviews in psychology and health. Int J Clin Health Psychol 13(1):49–57.  https://doi.org/10.1016/S1697-2600(13)70007-3 CrossRefGoogle Scholar
  36. Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: EASE, vol 8, pp 68–77Google Scholar
  37. Pignone M, Saha S, Hoerger T, Lohr KN, Teutsch S, Mandelblatt J (2005) Challenges in systematic reviews of economic analyses. Ann Intern Med 142 (12_Part_2):1073–1079CrossRefGoogle Scholar
  38. Reed D, Price EG, Windish DM, Wright SM, Gozu A, Hsu EB, Beach MC, Kern D, Bass EB (2005) Challenges in systematic reviews of educational intervention studies. Ann Intern Med 142(12_Part_2):1080–1089CrossRefGoogle Scholar
  39. Riaz M, Sulayman M, Salleh N, Mendes E (2010) Experiences conducting systematic reviews from novices’ perspective. In: Proceedings of the 14th international conference on evaluation and assessment in software engineering, EASE’10. http://dl.acm.org/citation.cfm?id=2227057.2227063, pp 44–53
  40. Sackett DL (2000) Evidence-based medicine: how to practice and teach. EBMGoogle Scholar
  41. Sjoberg DI, Dyba T, Jorgensen M (2007) The future of empirical methods in software engineering research. In: Future of software engineering, 2007. FOSE’07. IEEE, pp 358–378Google Scholar
  42. Skoglund M, Runeson P (2009) Reference-based search strategies in systematic reviews. In: Proceedings of the 13th international conference on evaluation and assessment in software engineering, EASE’09, pp 31–40Google Scholar
  43. Spasic I, Ananiadou S, McNaught J, Kumar A (2005) Text mining and ontologies in biomedicine: making sense of raw text. Brief Bioinform 6(3):239–251CrossRefGoogle Scholar
  44. Strauss A (1967) The discovery of grounded theory: strategies for qualitative researchGoogle Scholar
  45. Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br J Manag 14(3):207–222CrossRefGoogle Scholar
  46. Wohlin C (2014a) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, ACM, EASE ’14, pp 38:1–38:10Google Scholar
  47. Wohlin C (2014b) Writing for synthesis of evidence in empirical software engineering. In: 8th international symposium on empirical software engineering and measurement. ACM, p 46Google Scholar
  48. Wohlin C (2016) Second-generation systematic literature studies using snowballing. In: 20th International conference on evaluation and assessment in soft engineering. ACM, p 15Google Scholar
  49. Zhang H, Babar MA, Bai X, Li J, Huang L (2011a) An empirical assessment of a systematic search process for systematic reviews. In: 15th international conference on evaluation and assessment in soft engineering. IET, pp 56–65Google Scholar
  50. Zhang H, Babar MA, Tell P (2011b) Identifying relevant studies in software engineering. Inf Softw Technol 53(6):625–637CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of AlabamaTuscaloosaUSA

Personalised recommendations