On the pragmatic design of literature studies in software engineering: an experience-based guideline

Abstract

Systematic literature studies have received much attention in empirical software engineering in recent years. They have become a powerful tool to collect and structure reported knowledge in a systematic and reproducible way. We distinguish systematic literature reviews to systematically analyze reported evidence in depth, and systematic mapping studies to structure a field of interest in a broader, usually quantified manner. Due to the rapidly increasing body of knowledge in software engineering, researchers who want to capture the published work in a domain often face an extensive amount of publications, which need to be screened, rated for relevance, classified, and eventually analyzed. Although there are several guidelines to conduct literature studies, they do not yet help researchers coping with the specific difficulties encountered in the practical application of these guidelines. In this article, we present an experience-based guideline to aid researchers in designing systematic literature studies with special emphasis on the data collection and selection procedures. Our guideline aims at providing a blueprint for a practical and pragmatic path through the plethora of currently available practices and deliverables capturing the dependencies among the single steps. The guideline emerges from various mapping studies and literature reviews conducted by the authors and provides recommendations for the general study design, data collection, and study selection procedures. Finally, we share our experiences and lessons learned in applying the different practices of the proposed guideline.

Appendices

Appendix: A Study Workflow Templates

In this appendix, we provide selected workflow templates, which we inferred from experiences (Table 5) for simple reuse in research method descriptions of scientific papers. The provided templates can be used to inspire or shorten the description of research methods, which especially in conference papers consumes much precious space. For each model in subsequent sections, we provide a brief context description, an exemplary workflow, and textual description.

1.1 A.1 Template 1: 2 Researcher Workshop Model with Snowballing

This model addresses smaller literature studies in which just two researchers collaborate, thus, having no option to implement more comprehensive study selection procedures, such as majority votes. Our experience shows this model to be well-applicable in settings with up to approximately 50 papers, two senior or one senior and one junior researcher, and in distributed settings. Apart from an initial research objective and/or a set of research questions and a (small) set of reference publications, no extra entry conditions need to be fulfilled.

Figure 8 illustrates the basic workflow for this model including some notes emphasizing the most relevant points to be considered.

Fig. 8

Exemplary workflow for the 2 researcher workshop model with a snowballing-based preliminary study

The 2 Researcher Workshop Model with Snowballing is implemented as follows: Right in the beginning of the study, a snowballing-based preliminary study is conducted. For this pre-study, a set of reference papers is selected to lay the foundation for an (incremental) snowballing search. When the snowballing is done, the obtained papers are analyzed for keywords, which are used to construct the search queries for an automated database search. As the last preparation steps, the data sources of interest are selected and the inclusion and exclusion criteria are defined.

The data collection is performed (according to the search strategy, Section 2.2.1). After the search, the dataset is cleaned (Section 2.2.2), e.g., by a stepwise integration of individual datasets. The kick-off meeting is—on the one hand—closing the data collection and cleaning phase and—on the other hand—starts the study selection phase. In the kick-off meeting, both researchers reflect on all the criteria, inspect and prepare the dataset for the rating, and agree on a schedule. According to the procedure illustrated in Fig. 5, each researcher gets a copy of the dataset and carries out the individual rating. When the rating is done, both datasets are integrated and checked for consensus. In a rating workshop (or multiple workshops), both researchers iterate through the dataset discussing all items that are not yet decided to find an agreement. When the concluding integration is done, the study selection phase is closed and the result set is transferred to the main study (Section 2.4). For handing over the result set, a copy of the fully rated result set is created for archiving, and the actual result set is reduced, i.e., those dataset items that were rated as irrelevant for the main study are removed from the dataset so that only relevant data finds its way into the analysis.

1.2 A.2 Template 2: 3 Researcher Voting-only Model

This model addresses literature studies in which three researchers collaborate and implement a voting-based study selection procedure. Our experience shows this model to be well-applicable in the majority of all literature study settings. This model supports mixed and distributed teams, whereas at least one senior researcher has to be involved to guide the study project. Our standard implementation of the 3 Researcher Voting-only Model follows the 2+1 approach (Fig. 5, p. 15), i.e., the voting procedure to select relevant papers is organized by two researchers carrying out the full voting independently and calling in a third researcher to make the final decisions. In order to set up a study following this model, research objectives and questions, keyword lists and accordingly derived search queries have to be in place; optionally, a (small) set of reference publications is available.

Figure 9 illustrates the basic workflow for this model including some notes emphasizing the most relevant points to be considered.

Fig. 9

Exemplary workflow for a data collection and study selection approach for 3 reviewers using a voting-only approach

The 3 Researcher Voting-only Model is implemented as follows: After defining the search queries, data sources of interest, and the required inclusion and exclusion criteria, actual data collection is performed (Section 2.2.1). After the data collection, the data sets are cleaned (Section 2.2.2), e.g., via a stepwise integration of individual datasets.

In the kick-off meeting, the team of researchers nominates two researchers who will conduct the initial rating. According to the procedure illustrated in Fig. 5, each of the two selected researchers gets a copy of the integrated dataset for carrying out the individual rating. When both researchers have rated the dataset, one of them integrates both and analyzes the integrated result set for the agreement. Those dataset items that are not yet decided are selected and exported in a reduced dataset, which is given to the third reviewer. The third reviewer then performs a rating on the reduced dataset and, eventually, integrates the outcome with the full dataset. After performing this third rating, the dataset is now fully decided and can be prepared to be transferred to the main analysis (Section 2.4). If using a tool-supported approach as, for instance, shown in Fig. 10, the different stages can be supported by simple calculation, scripts, and conditional formatting (color coding).

Fig. 10

Example of a color-coded voting spreadsheet. The sheet shows different combinations of a 3-person majority vote (2 reviewers + 1 extra reviewer for final decisions)

Appendix: B Recommended Data Structure

In this section, we present a recommendation of a data structure to store data obtained by a manual/automatic literature search. Table 8 presents this recommended data structure, which emerges from several literature studies (Table 5), and the table explains the meaning of the different fields. Note: We consider the presented data structure to be minimal, i.e., specific studies will require further data fields. However, due to the absence of comprehensive and mature tools to support mapping studies, the normal would be to set up a simple spreadsheet. Examples of such spreadsheets (Fig. 10) can be obtained from http://goo.gl/PBylsn.

The data structure as presented in Table 8 only contains a minimal set of data, which needs to be extended according to the study’s scope. For systematic mapping studies, the following extra data should be contained:

• Generic/reused classification schemas, such as research/contribution type facet (Wieringa et al. 2005, Petersen et al. 2008)

• Study-specific classification schemas, such as focus type facets (Paternoster et al. 2014) or rigor/relevance models (Ivarsson and Gorschek 2011)

• In-/exclusion criteria to document, why a paper was in-/excluded (cf. Table 2)

Table 8 Recommended minimal data structure

Furthermore, grounded in our experience from Kuhrmann et al. (2015), we also recommend adding “dynamic metadata” to the data structure (as already mentioned in Table 8). Such metadata can be added on-the-fly and can support the enhancement of the dataset. From our experience (Kuhrmann et al. 2016), we recommend to collect metadata at least from the dimensions Study and Context.

The dimension Study covers the overall research approach followed in a particular paper, e.g., is a particular paper a primary study, a replication, or even a secondary study, and it can even contain the research methods used, such as interview research or grounded theory analyses. Metadata from this category supports a more detailed classification and analysis of papers regarding the research and contribution type facets. The dimension Context aims at collecting as much context information from the selected papers as possible, such as the software engineering lifecycle phase addressed by a paper (e.g., design, coding, test), the organizational context in which the research was conducted (e.g., SMEs, global players etc.), and the application domain of a paper (e.g., automotive software or software for the healthcare domain).