Software Quality Journal

, Volume 16, Issue 2, pp 237–261

Software process improvement in small and medium software enterprises: a systematic review

Authors

    • IDIS Research Group, Electronic and Telecommunications Engineering FacultyUniversity of Cauca
    • ALARCOS Research Group, Information Systems and Technologies Department, UCLM-INDRA Research and Development InstituteUniversity of Castilla-La Mancha
  • Félix García
    • ALARCOS Research Group, Information Systems and Technologies Department, UCLM-INDRA Research and Development InstituteUniversity of Castilla-La Mancha
  • Mario Piattini
    • ALARCOS Research Group, Information Systems and Technologies Department, UCLM-INDRA Research and Development InstituteUniversity of Castilla-La Mancha
Article

DOI: 10.1007/s11219-007-9038-z

Cite this article as:
Pino, F.J., García, F. & Piattini, M. Software Qual J (2008) 16: 237. doi:10.1007/s11219-007-9038-z
  • 1.2k Views

Abstract

Small and medium enterprises are a very important cog in the gears of the world economy. The software industry in most countries is composed of an industrial scheme that is made up mainly of small and medium software enterprises—SMEs. To strengthen these types of organizations, efficient Software Engineering practices are needed—practices which have been adapted to their size and type of business. Over the last two decades, the Software Engineering community has expressed special interest in software process improvement (SPI) in an effort to increase software product quality, as well as the productivity of software development. However, there is a widespread tendency to make a point of stressing that the success of SPI is only possible for large companies. In this article, a systematic review of published case studies on the SPI efforts carried out in SMEs is presented. Its objective is to analyse the existing approaches towards SPI which focus on SMEs and which report a case study carried out in industry. A further objective is that of discussing the significant issues related to this area of knowledge, and to provide an up-to-date state of the art, from which innovative research activities can be thought of and planned.

Keywords

Software process improvementSPISmall and medium software enterprisesSMEsSystematic Review

1 Introduction

Small and medium enterprises are a very important cog in the gears of the world economy. Another very significant economic activity nowadays, which has emerged, grown and become strong over the last two decades, is the software industry. It is a major economic force throughout many nations in the world. The software industry in most countries has an industrial backcloth, made up mainly of small and medium software enterprises—SMEs—(Fayad et al. 2000) which favour the growth of national economies. According to Fayad et al. (2000) 99.2% of software development companies are small and medium (fewer than 250 employees). They develop significant products, for the construction of which the firms need efficient Software Engineering practices that are suitable for their particular size and type of business.

From the very beginning of the 90s onwards, the Software Engineering community (industry and researchers) has expressed a special interest in Software Process Improvement (SPI). This is evidenced by the growing number of articles that deal with the topic of SPI, as can be seen in the analysis of the trends in publications on the subject of SPI, presented in Hall et al. (2002). Also worthy of note is the great number of international initiatives related to SPI, such as CMM, CMMI, ISO/IEC 15504:2004, SPICE (ISO/IEC 15504:1998), ISO/IEC 12207:2004 and ISO 9001:2000, among others.

Nevertheless, in educational and industrial environments there is a widespread tendency to emphasise that the success of SPI is, in most cases, only possible for large companies which have enough resources to tackle this type of practice. This perception is based on the fact that often SPI programmes are just not viable for SMEs, because of their organizational structure and the cost that is involved. This viewpoint also takes into account the reality that the models of improvement created by international organizations have not been considered for this type of companies, but rather for large-scale ones (Saiedian and Carr 1997). SEI (2006) and ISO (2006) have recently shown how the Software Engineering Institute, SEI, and the International Organization for Standardization, ISO, have each in turn begun to make efforts to ensure that their standards of software process improvement can be applied to SMEs. Many authors agree that the special characteristics of small companies mean that process improvement programmes must be applied in a way that is particular to them and visibly different from how this is done in the large organizations. This is not as simple as just regarding these programmes as scaled down versions of those which are applied in big companies (Richardson 2001).

In this article we present a systematic review of the literature which deals with the existing approaches of SPI in SMEs and which also include a case study within the industry. In accordance with the general goals of systematic reviews, our aim is to provide an up-to-date state of the art which synthesizes the work in this area of knowledge and can be used to identify gaps from which to formulate innovative research activities. The paper proceeds as follows. In Sect. 2 the systematic review itself is presented. Section 3 presents the results obtained and a discussion of them. Finally, the conclusions and future work are outlined.

2 Systematic review of SPI in SMEs

The systematic review which follows has been carried out in the context of the research project COMPETISOFT (Oktaba et al. 2007), whose aim is to provide small and medium enterprises in Latin America with a framework for the improvement of their process which is tailored to their special characteristics. The guidelines presented in Kitchenham (2004) and the protocol template presented in Biolchini et al. (2005, 2007) were followed. We have, moreover, looked at other similar systematic reviews (Hannay et al. 2007; Jørgensen and Shepperd 2007; Staples and Niazi 2007). All the templates of the systematic review of SPI in SMEs can be seen in Appendix A. In this section we set out an overview of the first four stages of the review protocol.

2.1 Question formularization

The research question is: What approaches concerning SPI exist, which focus both on SMEs and report on a case study? The list of keywords used to find an answer to the research question was: software, process, improvement, SPI, small, enterprises, organizations, companies, team, firms, SME, settings, CMM, CMMI, 15504, SPICE, 12207 and 9001.

The results expected at the end of the systematic review were, amongst others, to discover what the improved processes were, as well as to identify the key factors for process improvement. We also expected to see what improvement strategies had been carried out, along with what the most frequently-used models for the process improvement were.

The population group observed was the set of proposals concerning SPI that have been applied within SMEs, as published in the works in our list of sources. In the context of the present work and using the terms of reference given by (EU 2005) small enterprises are companies with fewer than 50 employees and medium enterprises are companies which have between 50 and 249 employees.

2.2 Sources selection

By taking the list of keywords seen above as a starting point, and making combinations with logical connectors “AND” and “OR”, two basic search strings were obtained (see Table 1). When the time came to carry out the searches, the basic search strings were adapted to each of the search engines of the respective sources.
Table 1

Basic search strings

Basic search strings

1

“software process improvement” AND (small AND(enterprises OR organizations OR companies OR team OR firms OR settings))

2

(small AND (enterprises OR organizations OR companies OR team OR firms OR settings)) AND (CMM OR CMMI OR 15504 OR SPICE OR 9001 OR 12207)

With the first string it is possible to find studies about SPI and SMEs that are related to some methods of software engineering (e.g. six-sigma, agile/XP). With the second string it is possible to find studies about SMEs which are related to specific methods of software process improvement. The basic search strings of Table 1 were not intended to search for specific results which contain empirical studies in the software industry (e.g. case studies). So, in line with our research question, we used the inclusion and exclusion criteria to select only the publications which reported case studies (quality threshold).

The planned list of sources with which the systematic review was carried out is:
  • Science@Direct, on the subject of Computer Science,

  • Wiley InterScience, on the subject of Computer Science,

  • IEEE Digital Library,

  • ACM Digital Library, and

  • As grey literature, the special report “Proceedings of the First International Research Workshop for Process Improvement in Small Settings” from the SEI was reviewed.

The sources have been identified on the basis of the judgement of experts in the area of work which is the scope of the systematic review. These sources include certain highly important journals in which our research area is widely dealt with, such as: Information and Software Technology, Software: Practice and Experience, Software Process: Improvement and Practice, IEEE Software, Software Technology and Engineering Practice, Computer, among others. Furthermore, we chose “Workshop for Process Improvement in Small Settings” because this is a specialised event organised by SEI, in which significant proposals for process improvement in small teams, small projects, small organizations, and small businesses, are presented. Given all that we have just said, a significant number of the works presented in these sources are more likely to be included in the primary studies list than are other non-specialized workshops on this topic.

2.3 Studies selection

We propose that the criterion for carrying out the systematic review be based on an iterative and incremental procedure. The term “iterative” indicates the repetition of one or more activities; the term “incremental” indicates that implementation proceeds from an initial subset of the sources to more and more complete subsets, until the whole review is addressed. It is iterative, because the execution (search, information extraction, and visualization of results) of the systematic review is carried out completely on one search source first, and then on the others in turn. It is also incremental in the sense that the document (which is the product) of the systematic review grows and evolves with each iteration until it becomes the definitive one.

In this way, many of the advantages that these types of development methods offer were obtained, for example:
  • Implementation starts from a source, reducing the complexity and scope of each execution.

  • Each cycle is short and provides feedback for the next cycle.

  • There is an integration of the results of each cycle into the final review.

  • This integration can be progressively achieved by successive releases of the review, each release achieving more primary studies.

  • The increments can be carried out both independently and in parallel.

  • The complexity of each increment is easier to administer.

The procedure for gathering information for the primary studies is illustrated in the activity flow diagram in Fig. 1. In order to manage the searches, we propose that from the analysis of primary studies a list with the new potential primary studies should be generated (secondary searches). Once the search is finished and the information has been obtained from selected sources, the studies of the former list must be subjected to the procedure for execution of the systematic review. This will obtain the new primary studies and then their information will be synthesized.
https://static-content.springer.com/image/art%3A10.1007%2Fs11219-007-9038-z/MediaObjects/11219_2007_9038_Fig1_HTML.gif
Fig. 1

Procedure for execution of the systematic review which obtains the primary studies and synthesizes their information

The inclusion criterion for what to include as regards studies took as its basis an analysis of the title, the abstract, and the key words from the articles obtained in the search. These served to determine whether or not they were related to SPI and if they focused on, and were applied to, SMEs. Another focus of analysis was how the key words improvement and small were dealt with in the total content of each article, in order to identify whether a given study presents a real case study. The goal was to decide whether or not a given article should be chosen as a relevant study in the context of the systematic review (in other words whether it was a potential candidate for primary study).

From the analysis conducted in the first iteration of the relevant studies, we ascertained that there were different levels of treatment for the SPI and SMEs:
  • A first level, in which the study consists of the application of a process improvement effort that focuses on small and medium-size organizations (there is a case study applied to SMEs, see all studies selected in Appendix B).

  • A second level, in which the study deals with SPI initiatives for SMEs which have nevertheless never been applied in such companies (where there is no case study applied to SMEs, for instance (Brodman and Johnson 1994) or where it is a theoretical study which is not based on direct/empirical observation).

  • A third level, in which the study reports on SPI initiatives where some SMEs are involved, but improvement in these companies was not the main objective of those initiatives (see, for example, the study presented in El Emam and Birk 2000).

To determine which articles, within the context of the systematic review, were important enough to be considered as primary studies, the exclusion criterion was to exclude any of those which dealt with SPI and SMEs at levels two and three.
After applying the procedure for the gathering of primary studies, 743 studies were found; 400 of these were non-repeated and from these, 45 primary studies were obtained (see Table 2).
Table 2

Distribution of studies by source (January 1996–March 2006)

Sources

Studies

%

Search date

Discovered

Not repeated

Relevant

Primaries

Science@Direct

06/03/2006

157

42

4

4

8.9

Wiley InterScience

21/03/2006

76

30

17

15

33.3

IEEE Digital Library

03/04/2006

208

128

22

19

42.2

ACM Digital Library

18/04/2006

272

170

7

1

2.2

Proceedings

30/04/2006

30

30

6

6

13.3

 

Total

743

400

56

45

100

In this systematic review potential references within primary studies were not searched for.

2.4 Information extraction

As can be observed in Fig. 1, once the primary studies were chosen, the extraction of the relevant information for the systematic review was carried out. The inclusion criterion for the information coming from the primary studies consisted of whatever gave an indication of what the improvement strategy used was like, along with information regarding the processes that had been improved, the key factors in the success of the improvement, and the models used for that improvement of the processes. Alongside the above information, the most important ideas of the study are registered. The information from the primary publications was stored on a chart in which the data extraction format was structured in the following columns: name of the study, magazine, date, authors, methodology followed, results, objectives of the study, country where the study was carried out, number of SMEs involved, number of employees in the SMEs involved, information related to the improvement models, the improvement proposal, key factors in the success of a SPI programme, the improvement processes, and the subjective results of the study (Table 3).
Table 3

Data extraction format

Data extraction from the study

Name

Practical Software Process Improvement—The IMPACT Project.

Journal/Conference

13th Australian Software Engineering Conference (ASWEC’01).

Date

August 2001.

Authors

Louise Scott, Ross Jeffery, Lucila Carvalho, John D’Ambra, Philip Rutherford.

Objective results of the study

Methodology

Case study

Outcomes

• This paper presents a framework for SPI that is designed to utilise a range of improvement technologies and which supports continuous and highly focused improvement over many projects, thus producing timely, cost-effective and tangible improvements for SMEs.

• The effectiveness of the framework is illustrated in this paper with its application in a small, Sydney-based, web-development company.

Country

Australia

No. SMEs

1

No. Employees

20

Models/Standards

IMPACT, ISO 12207

Improvement proposal

Uses an improvement framework in order to guide the programme SPI.

Success factors

Guide the improvement programme. Have got freedom in choosing the improvement technology.

Improvement processes

Projects documentation, process planning, process estimation and requirements elicitation.

Subjective results of the study

• The framework is cyclic—advocating an “understand—improve—apply—measure” approach which can be applied incrementally across many projects. On each cycle a different focus for improvement is chosen so that each iteration need not require a large expenditure on the part of the company and the improvements are available to all subsequent projects. By implementing many small iterations of the improvement cycle results are achieved quickly (within a few months) and are measurable in the projects in which improvement has been applied. The framework is implemented using existing and proven technologies, frameworks and standards that can be purchased (at a greater or lesser expense depending on the choice of the company) commercially.

• Accordingly, our framework for software process improvement, as shown in Fig. A, differentiates between the project level and the process level. At the project level, many projects are carried out according to good project management practice (e.g. a “Plan–Do–Check–Act” cycle). This is shown as the small cycle in Fig. A which we call the project cycle. At the process level, experience and understanding obtained from many projects are used to understand and improve the generic process model which is then used to guide future projects. This is the large cycle shown in Fig. A which we call the process cycle. Note that these cycles interact closely—the process cycle drives the execution of the projects and the projects drive improvements to the process. The following paragraphs will expand further on each of these cycles and how they influence software process improvement.
https://static-content.springer.com/image/art%3A10.1007%2Fs11219-007-9038-z/MediaObjects/11219_2007_9038_Figd_HTML.gif
Fig. A

A framework for software process improvement

During the information extraction phase we assessed the quality of each primary study by means of the attributes: “Bias”, “Internal Validity” and “External Validity”. Each attribute was scored “Yes” or “No” depending upon whether the study explored explicitly the possibility of biases (selection, experimenter or publication) or threats (internal or external validity). From the 45 primary studies none scored “YES” for all these attributes. Seven studies mentioned at least bias issues (primary studies 2, 3, 7, 22, 34, 35, 38—Appendix B). Only two studies mentioned internal and external validity (primary studies 5 and 12). These results may be due to the nature of studies, as they were case studies, and especially the fact that the treatment of internal and external validity threats takes place in the context of formal experiments.

Furthermore, a study design assessment was performed in order to obtain greater insight into the quality of primary studies. The type of design of each study was extracted and classified according to the study design hierarchy for software engineering presented in Kitchenham (2004). With regard to the quality of the publications (from this point of view) included in the systematic review it can be stated that primary studies are observational ones with design type case series, by means of which the formulated improvement proposals were applied to SMEs. From these design type case series, 82% are post-test and 18% are pre-test post-test. Additionally, all studies show a qualitative analysis of the results obtained, whereas just 27% of the studies present a quantitative analysis of some of the results and only 47% of the studies obtained evidence from 2 or more SMEs.

3 Results and discussion

In this systematic review, we have formulated a research question which aims to obtain insight into the current state of practice of SPI efforts in industry. The objective is to use the current state of the art to identify new research activities to carry out in the field of SPI.

Before going into the main results in depth, we should first note that there is an increasing interest in tackling the area of SPI and SMEs on the part of the Software Engineering community. This is indicated by the appearance of a great number of standards and proposals related to SPI for SMEs, and also by the increasing number of studies that deal with the subject, according to the analysis of the trend in publications that was carried out (see Fig. 2). The countries which report most improvement efforts in SMEs are, in descending order, Australia, Ireland, Finland, Denmark, Brazil, USA, Sweden and Mexico. Several of the studies reported on come within the framework of strategies encouraged by governmental organizations. Together with all of the above considerations, company commitment is vital (Dyba 2005), as is a firm belief on the company’s part that starting an SPI project will give it competitive advantages.
https://static-content.springer.com/image/art%3A10.1007%2Fs11219-007-9038-z/MediaObjects/11219_2007_9038_Fig2_HTML.gif
Fig. 2

SPI publication trends in SMEs

A statistical analysis was conducted with the information extracted from the studies, whose purpose was to show the relevant findings of the systematic review that had been carried out. These relevant findings are based on the discussion of significant aspects related to this area of knowledge and include aspects such as size of the companies involved, models used, improvement proposals, processes improved and factors in success.

3.1 Companies involved

The average number of SMEs involved per study is 2.97. This means that in each of the studies an average of three companies is used. We should bear in mind, however, that 53% of the studies involved only one company.

Currently, SPI efforts are also being applied to small companies (see Fig. 3). In this figure it can be seen that 47% of the companies involved in the improvement efforts were companies with a number of employees of between 10 and 49 and 33% were companies with fewer than 10 employees, which gives us the sum of 80% of all the companies in which an SPI effort was reported to have been carried out. Only 20% is reported in medium sized companies (between 50 and 249 employees).
https://static-content.springer.com/image/art%3A10.1007%2Fs11219-007-9038-z/MediaObjects/11219_2007_9038_Fig3_HTML.gif
Fig. 3

Number of employees in the companies where an improvement effort project was conducted

3.2 Models used

Within the Software Engineering community there is a growing concern that SEI and ISO standards are not easily applicable to small firms because they require a huge investment (in time, money, and resources) (Saiedian and Carr 1997). Table 4 presents the standards used by each study. The SMEs adapt and use these models to initiate their improvement efforts, however. The most widely-used SEI model is CMM, with a 25% share. The SEI models have been used in 51% of the reported improvement efforts. The most extensively-used ISO models are the 15504 (SPICE) and the 9001, at 9% each. The ISO/IEC 15504 model has the two-fold nature of being an evaluation method and a process model. The ISO models have been used in 31% of the improvement efforts carried out; 18% of the improvement efforts reported in the studies do not use either a SEI or an ISO model.
Table 4

Standards used by each study

Standards

Studies

Total

%

CMM

11, 14, 16, 18, 25, 26, 28, 29, 30, 32, 33, 26, 39, 40, 41, 42, 44

17

38

CMMI

17, 40, 42, 45

4

9

IDEAL

12, 18, 26, 39, 40, 42

6

13

ASSESSMENT SEI

14, 17, 26, 39, 40, 42, 45

7

16

15504 (SPICE)

2, 6, 9, 13, 21, 35

6

13

15504:2004

20, 41

3

7

12207

3, 4, 20, 22, 38

5

11

9001

11, 15, 28, 30, 33, 41

6

13

GQM

9

1

2

PSM

13

1

2

Others

1, 10, 22, 27, 41

5

11

Neither

8, 19, 23, 31

4

9

On the other hand, it is significant to highlight that in an improvement programme different types of models are involved. These include the model that guides the process improvement, the process assessment method and the process reference model that is to be followed. An adaptation of the IDEAL model is the means that is most widely used to guide improvement. The study shown in Casey and Richardson (2004) is a good example of this. IMPACT (Scott et al. 2001) and MESOPYME (Calvo-Manzano et al. 2002) models are also used, along with others. This type of model was used in nine studies, which is to say in 20% of the studies; this represents a low percentage, because an improvement model is the guide which is needed to articulate all the activities related to the improvement, and of course all the other models involved. In Fig. 4 the process assessment methods and process reference models used in the study are shown.
https://static-content.springer.com/image/art%3A10.1007%2Fs11219-007-9038-z/MediaObjects/11219_2007_9038_Fig4_HTML.gif
Fig. 4

Assessment and process models of the SEI and ISO involved in the improvement

As can be seen in Fig. 4, the assessment method that is most frequently used in the SMEs is the ISO/IEC 15504, in its 1998 and 2004 versions. Furthermore, the SEI assessments, the CBA-IPI proposals, or less intensive assessments such as the C class or micro-appraisal, follow in order of importance. We could add that the ISO 15504:2004 standard is beginning to be applied on a wide scale, due mainly to the fact that it is a model which is easy to understand, flexible, and fits the needs of SMEs. In some studies, the use of assessment models developed particularly for small companies and based on the ISO 15504 standard is shown, as for example in MARES (Wangenheim et al. 2006) and RAPID (Rout et al. 2000).

With regard to process reference models, that which is most widely used is the CMM (which is expected to be replaced in use by CMMI) of the SEI. In a CMM environment, the SMEs set out to achieve level 2 when an improvement process begins. It is important to underline that from the 122 companies involved in the primary studies, only two medium-size companies of 70 and 150 employees achieved the formal assessment as CMM-SW level 2. We consider that the motivation for these types of companies should therefore be to improve their processes in order to make them more effective and efficient units, rather than to gain the certification itself. Following these processes should be a means to an end and not an improvement goal (Paulk 1998).

Furthermore, from the analysis of the models used, we noticed that 30% of the improvement efforts which used CMM also used ISO 9001. If an ISO 9001-certified organization wishes to continuously improve its processes, the adoption of CMMI would be a good choice, since it provides more detailed practices for process improvement than ISO 9001 (Yoo et al. 2006). However, it is important to note that the compliance with ISO 9001 allows organizations to have broader beneficial effects which are beyond the scope of coverage of the CMM. Although ISO 9001 is a generic standard for quality management (and is therefore not directly concerned with software engineering best practices), we believe that it has been relevant for the software industry as it is more feasible to adopt (in cost and time) that others standards, especially for small companies.

Studies such as Hareton and Terence (2001), Johnson and Brodman (1999) and Saiedian and Carr (1997) show that the models from SEI or ISO are difficult for small and medium organizations to apply because of the complexity of their recommendations and the consequential large investment in terms of time and resources. From the results extracted it can be observed, nevertheless, that these models are the most widely applied in these kinds of companies. It is thus important to consider these models as the reference to develop research works about new models related with the tailoring and adaptation of SPI for SMEs. These new models would allow SMEs to obtain certification in the above-mentioned reference models as they grow in size and resources.

It is significant to underline that in most cases the improvement programmes in small software companies did not lead to the ISO or SEI certifications. The question which arises from this is whether software process improvement standards are suitable for small organizations. From the extracted primary study results and in accordance with the previously exposed paragraphs we consider that these standards per se are not suitable. The reasons why small organizations do not adopt these heavyweight approaches, such as for example CMMI, have been explored in Staples et al. (2007). This reinforces the need to have available strategies for process improvement which are tailored to small companies’ characteristics. These strategies must be aligned with the widely recognised standards in order to enable small companies to establish a solid basis for improvement which will lead them towards certification in these standards in the future. Furthermore, organizations which develop international Software Engineering standards should target the small enterprises separately from the medium enterprises. For instance the WG24 (an ISO/IEC JTC1 SC7 working group) has recently been established to develop international standards profiles in order to help apply them in very small enterprises (fewer than 25 employees).

According to Derniame et al. (1999), the emphasis on processes and on process management provides the main justification for many standardization initiatives of organizations such as SEI or ISO. It is also important to emphasize that process improvement is an integral responsibility of software process management (Florac et al. 1997). Consequently, we believe that it is important to work on how to establish a basic process management infrastructure that is tailored to SMEs and which will support the key principles of process management (to define, to measure, to control and to continually improve the processes). In this sense, and with regard to the field of process measurement, it is important to highlight that from the 45 studies that were analysed, 20 have used the SEI models, 18 have used ISO models and only 2 have used models for the definition of measurement programmes. This tends to support the assumption that the efforts of SMEs are still insufficient in the establishment of formal software measurement processes. Improvements introduced by the SPI are measured by means of informal and non-objective processes based on the employees’ perception and not through formal measurement programmes. This implies the need to dedicate more research effort to software measurement in SMEs, given that this is a core activity in the area of process management, which is a prerequisite to carrying out a suitable control and improvement of the processes.

3.3 Improvement proposals

The improvement proposals that have been suggested to guide the improvement effort are diverse. In Table 5 the improvement proposals found in this study are set out.
Table 5

Improvement proposals

% of Studies

Improvement proposals

Target

18

IP_1. Establish software processes (Use of electronic process guides and experience repository—EPG-ER; Adaptation and use of practices RUP, XP, SCRUM, among others; Informal self-assessment for software developers (web tool))

The software process

9

IP_2. Prioritize the SPI efforts (by means of DAIIPS method, Software Process Matrix, Express Process Appraisal or Framework of key decisions of business strategy and product development)

The SPI programme

4

IP_3. Evaluation of a SPI programme (with low-rigour, or to define and use a metrics programme)

13

IP_4. To guide the SPI efforts (Primer method, Framework IMPACT, MESOPYME method/model, using improvement patterns, following an assessment process, neuronal network)

45

IP_5. Adaptation and use of SPI models/methods (PSP, TSP, CMM, CMMI, IDEAL, ISO 15504:2004, SPICE, ISO 9001)

11

IP_6. Others (Definition and use of test framework; Lead process improvement experiment; Use knowledge management for SPI; Acquire ready-to-use technical infrastructure; Improve the relationship and cooperation with clients)

With the aim of providing an initial basis for classification of improvement strategies which synthesize the obtained results, six exclusive categories have been identified (from IP1 to IP6), which are shown in Table 5. These categories were defined by means of an in-depth analysis and subsequent synthesis of commonalities and differences of the SPI proposals extracted from each primary study. First of all, each publication was thoroughly analysed and then a summary of the main ideas and contributions was extracted. The next step was to group the SPI contributions which were related to a more general concept into categories, which resulted in the six exclusive categories. For example, the improvement proposals “to use an improvement framework in order to guide the SPI programme” (see Table 3) and “to follow the concept of action packages in order to carry out the implementation of the SPI in SMEs” (Calvo-Manzano et al. 2002) can be grouped under the category IP4 “to guide improvement efforts”. These categories may evolve in future replications of this systematic review.

The improvement proposals focus on two targets: the software process and the SPI programme. From Table 5, we can observe that what most improvement proposals (71%) have concentrated on is the guidance of an improvement project and the priority of improvement implementation, along with the use of existing improvement models, adjusting them to needs. They have also put great emphasis on the evaluation of the improvements introduced for the SPI program. Alongside this greater bulk of proposals, there are others (18%) which focus on the definition, evaluation, and support of the software processes.

From the analysis of the primary studies extracted, it can be concluded that the improvement approaches can be generated at an organizational or a technical level. It should be emphasized that there are studies (11%) which, in the quest to support actions of process improvement, propose the supplying of technical tools and facilities that support the activities related to the processes. This is termed as technical infrastructure and includes features such as document templates, process models, techniques and tools.

3.4 Improved processes

The primary studies were used to search for the information that reported on the improved processes in an explicit way. All studies expressed in natural language some type of process improvement. This was stored in the data extraction forms described in Sect. 2.4. After that, this information relating to improvement, expressed in natural language, was analysed and adapted, the aim being to represent it in a process model. This allowed us to have a vision of what the improvement efforts in small and medium size companies had focused on. The process model used to express the improvements was the ISO/IEC 12207:2004 (ISO 2004). The frequency of the improved processes is presented in Fig. 5. Improvement efforts, as is clear from Fig. 5, aim to improve processes such as project management, documentation, requirement change management, process establishment, configuration management and requirement elicitation. There are other processes, as is the case of joint review, which do not report any type of improvement.
https://static-content.springer.com/image/art%3A10.1007%2Fs11219-007-9038-z/MediaObjects/11219_2007_9038_Fig5_HTML.gif
Fig. 5

Process improvement by improvement efforts in SMEs

3.5 Success factors

The primary studies were also used to search for and extract the information that reported the practical advice (success factors) in the success of SPI projects. These success factors were analysed and synthesized, resulting in the following:
  • Initiate the improvement as soon as possible with a simple structure of the SPI model, but before starting the SPI programme ensure that the organisation is sufficiently stable. It is important to have expert consultantship when beginning the SPI programme.

  • Guide the improvement programme, following a systematic and coherent initiative by means of specific procedures and combining different approaches. This procedure must follow an iterative and incremental approach (prioritize the improvement points defined by the organization) that allows a continuous adoption of improvement practices.

  • Provide fast return on investment and yield advances to the organization, carrying out improvements that have a maximum effect in as short a time as possible. Furthermore, get optimum value out of resources, stretching them as far as they will go.

  • Look for external financial aid or join together with other companies so as to both finance and share specialised resources that are involved in improvement.

  • Involve the greatest possible number of company employees, using personnel training to ensure as few interruptions as possible in the SPI programme.

  • Minimize resistance to change, through organizational awareness of how the improvement, if put into operation as a response to a real need, will greatly benefit the company.

  • Track and supervise the SPI programme by means of frequent assessments of processes, in order to evaluate the efficiency of this programme. A good strategy is to carry out rapid assessments of software processes.

  • Tailor and carry out measurement activities by means of the systematic use of process metrics.

  • Establish the infrastructure that supports an efficient communication between the different actors involved in the improvement.

  • The managerial commitment and their strategic plan of the organization are fundamental within an SPI programme. There must be a commitment to an ongoing quest for quality on the part of all the organization’s employees.

  • Tackle the problem of improvement from the technical perspective.

3.6 Limitations

The systematic review presented in this paper has some limits which should be taken into account if we are to understand to what extent the results can be generalized and how they can be used:
  • The review has been performed in the context of the COMPETISOFT project. So, as the group of researchers is small (a researcher and two supervisors), the scope of the revision had to be well defined and therefore, for the first execution of the systematic review procedure the secondary searches were not taken into account.

  • The observations and conclusions presented are based on the results extracted from 45 primary studies, which is a significant and representative sample of the published works on the topic, in accordance with the quality of the selected sources. Systematic reviews, however, do limit the scope of the population mainly to that of published studies, which can be a relatively low percentage of the overall population. This can limit the power of generalization.

  • The selected studies include empirical validation based on case studies and all of them presented positive results. Furthermore, because of the nature of primary studies, it was difficult to find quality attributes in the context of case studies. Our assessments about quality attributes are not judgements about whether the primary study had a “good” treatment of these issues (it is limited only to analyse whether the studies considered these issues).

  • A sensitivity analysis was not applied: (i) with regard to the inclusion of only high quality studies, given the low number and proportion of primary studies which dealt with quality attributes; and (ii) given the heterogeneous nature of primary study results which made the quantitative comparison difficult and the carrying-out of meta-analysis unfeasible (additionally, there are no standardized/universally accepted criteria for reporting results in the field of SPI).

4 Conclusions and future work

Systematic reviews require an effort which is considerably more intensive than that needed in the conventional reviews of literature. The formality with which the systematic review is carried out allows us to validate any results reported in the systematic review which are, in turn, both supported and guaranteed by the review protocol.

In this article a systematic review of the literature on SPI efforts carried out in SMEs has been presented, which has allowed us to obtain a view about SPI in SMEs, using a representative set of related works which was obtained as a result of the systematic review.

From the results obtained in the current review, the first observation from the study that was carried out is that, in the last 5 years, the Software Engineering community has shown an ever-increasing interest in tackling SPI in SMEs. Although at a global level there is a general consensus that SEI and ISO models can be applied only with difficulty in small companies, these same companies do indeed adapt and use these models when undertaking their improvement efforts. Small companies are currently undertaking SPI efforts, but SEI or ISO models do not lead to these companies receiving their certification. The results presented in this paper show that it is indeed very difficult to successfully apply formal SPI programmes which use models such as, for example CMM, to SMEs.

Another relevant fact is that the most frequently used improvement models in the SMEs are CMM (which is expected to be replaced in use by CMMI) as a process reference model, ISO 15504 as a process assessment method and IDEAL as the model for guiding improvement. Nevertheless, there are no studies in which the application of the three models to a small company is reported. It is important to notice that the widespread use of CMM is due to the fact that it was one of the first models on the market and was heavily promoted by the DoD in the USA.

The vast bulk of improvement proposals major on guiding the improvement project, prioritizing SPI implementations, using existing improvement models and fitting them to SMEs needs, as well as on an evaluation of improvement brought about by an SPI programme. Improvements introduced by SPI efforts are, however, measured through informal and non-objective processes, based on the employees’ perception and not through formal measurement processes. There is a smaller percentage of other improvement proposals which have as a focal point the definition, assessment and support of the software process. The focus is also to undertake improvement actions whose fundamental aim is to provide technical infrastructure to support activities related to the processes. Furthermore, as is shown in Fig. 5, improvement efforts aim to improve processes such as project management, documentation, requirement change management, process establishment, configuration management and requirement elicitation. These processes are related to the support, management and improvement of processes.

Using the protocol of this systematic review as a starting point, secondary searches from primary studies and new sources of studies (e.g. Springer or conferences related to SPI) will be reviewed in future iterations, following the suggested procedure for execution of the systematic review. Furthermore, studies which deal with SPI initiatives for SMEs, but which do not report a real case study, will also be analysed. These kinds of studies have been excluded from the present work, as they did not come within its scope.

Acknowledgments

We would like to thank anonymous reviewers for their useful comments in order to improve this paper, and we acknowledge the assistance of the following projects: COMPETISOFT (506AC287) financed by CYTED, MECENAS (PBI06-0024) granted by the “Junta de Comunidades de Castilla-La Mancha” of Spain and ESFINGE (TIN2006-15175-C05-05) financed by Dirección General de Investigación of the Ministerio de Educación y Ciencia of Spain.

Copyright information

© Springer Science+Business Media, LLC 2007