Software process improvement in small and medium software enterprises: a systematic review

Abstract

Small and medium enterprises are a very important cog in the gears of the world economy. The software industry in most countries is composed of an industrial scheme that is made up mainly of small and medium software enterprises—SMEs. To strengthen these types of organizations, efficient Software Engineering practices are needed—practices which have been adapted to their size and type of business. Over the last two decades, the Software Engineering community has expressed special interest in software process improvement (SPI) in an effort to increase software product quality, as well as the productivity of software development. However, there is a widespread tendency to make a point of stressing that the success of SPI is only possible for large companies. In this article, a systematic review of published case studies on the SPI efforts carried out in SMEs is presented. Its objective is to analyse the existing approaches towards SPI which focus on SMEs and which report a case study carried out in industry. A further objective is that of discussing the significant issues related to this area of knowledge, and to provide an up-to-date state of the art, from which innovative research activities can be thought of and planned.

Appendices

Appendix A

The protocol summary for this systematic review is presented below. To carry out the systematic review of SPI in SMEs the protocol template presented in Biolchini et al. (2005) and Kitchenham’s guidelines for systematic reviews presented in Kitchenham (2004) were followed, along with some advice from other published systematic reviews.

1.1 Review planning

1.1.1 Question formulation

Question focus The goal of this systematic review is to identify experience reports and initiatives carried out in Software Engineering related to software process improvement in SMEs.

Question quality and amplitude

• Problem: The improvement models established by the SEI or ISO were structured in such a way as to be applicable to large companies. These models cannot easily be applied in small and medium organizations, given the fact that an improvement project involves a large investment in terms of time, money and resources. The great complexity of the recommendations must also be taken into account, along with the fact that the return on the investment undertaken has to be seen from a long term perspective (Hareton and Terence 2001; Johnson and Brodman 1999; Saiedian and Carr 1997).

• Question: What approaches concerning SPI exist, which focus both on SMEs and report on a case study?

• Keywords and synonyms: software, process, improvement, SPI, small, enterprises, organizations, companies, team, firms, SME, settings, CMM, CMMI, 15504, SPICE, 12207 and 9001.

• Intervention: To analyse/study the different proposals about software process improvement which have been applied on SMEs.

• Control: There are no initial data for this systematic review.

• Effect: Identification of the processes improved, key factors for improvements, improvement strategies conducted and the models for process improvement which are most usually applied in SMEs.

• Outcome: To present an up-to-date state of the art regarding SPI proposals applied in SMEs from which to think of and plan innovative research activities as well as to discuss the significant issues related to this area of knowledge.

• Population: Research proposals in the software processes improvement area which have been applied within SMEs and which have been published in the list of sources.

• Application: People related to SPI and SMEs, such as instructors, researchers, company managers and improvement project managers, among others.

• Experimental design: A non-statistical method will be applied.

1.1.2 Sources selection

Definition of source selection criteria The sources have been identified based on the judgment of experts in the area of work which is the scope of the systematic review.

Study language English

Source identification

• Source search method: The search execution of the primary results will be performed by means of the search web engines of each of the selected sources.

• Search string: There are two strings.

  • First “software process improvement” AND (small AND(enterprises OR organizations OR companies OR team OR firms OR settings)).

  • Second (small AND (enterprises OR organizations OR companies OR team OR firms OR settings)) AND (CMM OR 15504 OR SPICE OR 9001 OR 12207).

• Source list: The planned list of sources is:

  • Science@Direct on the subject of Computer Science,

  • Wiley InterScience on the subject of Computer Science,

  • IEEE Digital Library,

  • ACM Digital Library, and

  • As grey literature the special report “Proceedings of the First International Research Workshop for Process Improvement in Small Settings” from SEI was reviewed.

• Source selection after evaluation: Expert opinion concluded that, a priori, all the selected sources satisfy the quality criteria, as the sources include significant and highly important journals in the area of knowledge tackled by the systematic review.

• Reference checking: The sources were approved.

1.1.3 Study selection

Study definition

• Study inclusion and exclusion criteria definition: Since search strings are not intended to search empirical studies in the software industry, we defined the inclusion and exclusion criteria in such a way as to include publications that report case studies.

From the analysis conducted in the first iteration of the relevant studies, we ascertained that they treated the SPI and SMEs at different levels:

• A first level, where the study consists of the application of a process improvement effort that focuses on small and medium-size organizations (there is a case study applied to SMEs).

• A second level, where the study deals with SPI initiatives for SMEs which nevertheless have never been applied in such companies (there is no case study applied to SMEs).

• A third level, in which the study reports on SPI initiatives where some SMEs are involved, but improvement in these companies was not the main objective of those initiatives.

To determine which articles, in the context of the systematic review, were important enough to be considered as primary studies, the exclusion criterion was to exclude any of those which dealt with SPI and SMEs at levels two and three.

• Study type definition: Publications about SPI initiatives applied in SMEs, which have been reported as papers in the selected sources.

• Procedures for study selection: see Fig. 1.

1.2 Planning evaluation

The protocol evaluation of this systematic review was performed through:

• Protocol execution testing on one source in the first iteration of the review. During this iteration the protocol that was initially planned was evaluated and improved in order to achieve the review objectives, giving the final protocol presented in this work as a result. Some pre-searches were carried out to select the most suitable set of keywords and search strings. For example, in the planning evaluation stage and with regard to keywords, we initially considered the words “Small” and “Medium”, as small and medium are part of the acronym SME. However, when the search of this first iteration was performed by using the words “small” and “medium”, it was observed that the set of publications was a subset of the publications returned by using only the word “small”. Because of this, only the word “small” was included in the definitive search strings. The same strategy was followed when choosing the search strings in singular and/or plural forms. Finally, the plural form was considered to be most suitable to perform the searches as with singular search strings only a subset of publications extracted with strings in plural were obtained.

• Periodical meetings took place with two experts in the field of research, in order to validate the protocol and to track the review execution, to ensure that it was being carried out in a suitable manner.

1.3 Review execution

1.3.1 Selection execution

• Initial study selection: The complete selection execution and list of studies can be found in Pino et al. (2006).

• Study quality evaluation: See Sect. 2.4. Moreover, all selected studies have an assumption of quality, as they have been published in the selected sources. This means that the publications accepted have gone through a strict reviewing period which guarantees their quality.

• Selection review: The study selection was approved in control meetings with the experts in the area.

1.3.2 Information extraction

Information inclusion and exclusion criteria definition The data inclusion criteria from the primary studies consisted of collecting information about the improvement strategy applied, the processes improved, the improvement success key factors and the models used for process improvement. Moreover, the most important ideas from the studies were registered.

Data extraction forms See Sect. 2.4.

Extraction execution See Table 3. The complete data extraction execution can be found in Pino et al. (2006).

Resolution of divergence among reviewers There was no divergence; only one person carried out the execution of the review and two experts validated the information.

Appendix B: List of selected studies

In this section the selected primary studies in the systematic review are presented.

Table B.1 Primary studies selected in the systematic review