Software process improvement in small and medium software enterprises: a systematic review
- First Online:
- Cite this article as:
- Pino, F.J., García, F. & Piattini, M. Software Qual J (2008) 16: 237. doi:10.1007/s11219-007-9038-z
- 1.2k Views
Small and medium enterprises are a very important cog in the gears of the world economy. The software industry in most countries is composed of an industrial scheme that is made up mainly of small and medium software enterprises—SMEs. To strengthen these types of organizations, efficient Software Engineering practices are needed—practices which have been adapted to their size and type of business. Over the last two decades, the Software Engineering community has expressed special interest in software process improvement (SPI) in an effort to increase software product quality, as well as the productivity of software development. However, there is a widespread tendency to make a point of stressing that the success of SPI is only possible for large companies. In this article, a systematic review of published case studies on the SPI efforts carried out in SMEs is presented. Its objective is to analyse the existing approaches towards SPI which focus on SMEs and which report a case study carried out in industry. A further objective is that of discussing the significant issues related to this area of knowledge, and to provide an up-to-date state of the art, from which innovative research activities can be thought of and planned.
KeywordsSoftware process improvementSPISmall and medium software enterprisesSMEsSystematic Review
Small and medium enterprises are a very important cog in the gears of the world economy. Another very significant economic activity nowadays, which has emerged, grown and become strong over the last two decades, is the software industry. It is a major economic force throughout many nations in the world. The software industry in most countries has an industrial backcloth, made up mainly of small and medium software enterprises—SMEs—(Fayad et al. 2000) which favour the growth of national economies. According to Fayad et al. (2000) 99.2% of software development companies are small and medium (fewer than 250 employees). They develop significant products, for the construction of which the firms need efficient Software Engineering practices that are suitable for their particular size and type of business.
From the very beginning of the 90s onwards, the Software Engineering community (industry and researchers) has expressed a special interest in Software Process Improvement (SPI). This is evidenced by the growing number of articles that deal with the topic of SPI, as can be seen in the analysis of the trends in publications on the subject of SPI, presented in Hall et al. (2002). Also worthy of note is the great number of international initiatives related to SPI, such as CMM, CMMI, ISO/IEC 15504:2004, SPICE (ISO/IEC 15504:1998), ISO/IEC 12207:2004 and ISO 9001:2000, among others.
Nevertheless, in educational and industrial environments there is a widespread tendency to emphasise that the success of SPI is, in most cases, only possible for large companies which have enough resources to tackle this type of practice. This perception is based on the fact that often SPI programmes are just not viable for SMEs, because of their organizational structure and the cost that is involved. This viewpoint also takes into account the reality that the models of improvement created by international organizations have not been considered for this type of companies, but rather for large-scale ones (Saiedian and Carr 1997). SEI (2006) and ISO (2006) have recently shown how the Software Engineering Institute, SEI, and the International Organization for Standardization, ISO, have each in turn begun to make efforts to ensure that their standards of software process improvement can be applied to SMEs. Many authors agree that the special characteristics of small companies mean that process improvement programmes must be applied in a way that is particular to them and visibly different from how this is done in the large organizations. This is not as simple as just regarding these programmes as scaled down versions of those which are applied in big companies (Richardson 2001).
In this article we present a systematic review of the literature which deals with the existing approaches of SPI in SMEs and which also include a case study within the industry. In accordance with the general goals of systematic reviews, our aim is to provide an up-to-date state of the art which synthesizes the work in this area of knowledge and can be used to identify gaps from which to formulate innovative research activities. The paper proceeds as follows. In Sect. 2 the systematic review itself is presented. Section 3 presents the results obtained and a discussion of them. Finally, the conclusions and future work are outlined.
2 Systematic review of SPI in SMEs
The systematic review which follows has been carried out in the context of the research project COMPETISOFT (Oktaba et al. 2007), whose aim is to provide small and medium enterprises in Latin America with a framework for the improvement of their process which is tailored to their special characteristics. The guidelines presented in Kitchenham (2004) and the protocol template presented in Biolchini et al. (2005, 2007) were followed. We have, moreover, looked at other similar systematic reviews (Hannay et al. 2007; Jørgensen and Shepperd 2007; Staples and Niazi 2007). All the templates of the systematic review of SPI in SMEs can be seen in Appendix A. In this section we set out an overview of the first four stages of the review protocol.
2.1 Question formularization
The research question is: What approaches concerning SPI exist, which focus both on SMEs and report on a case study? The list of keywords used to find an answer to the research question was: software, process, improvement, SPI, small, enterprises, organizations, companies, team, firms, SME, settings, CMM, CMMI, 15504, SPICE, 12207 and 9001.
The results expected at the end of the systematic review were, amongst others, to discover what the improved processes were, as well as to identify the key factors for process improvement. We also expected to see what improvement strategies had been carried out, along with what the most frequently-used models for the process improvement were.
The population group observed was the set of proposals concerning SPI that have been applied within SMEs, as published in the works in our list of sources. In the context of the present work and using the terms of reference given by (EU 2005) small enterprises are companies with fewer than 50 employees and medium enterprises are companies which have between 50 and 249 employees.
2.2 Sources selection
Basic search strings
Basic search strings
“software process improvement” AND (small AND(enterprises OR organizations OR companies OR team OR firms OR settings))
(small AND (enterprises OR organizations OR companies OR team OR firms OR settings)) AND (CMM OR CMMI OR 15504 OR SPICE OR 9001 OR 12207)
With the first string it is possible to find studies about SPI and SMEs that are related to some methods of software engineering (e.g. six-sigma, agile/XP). With the second string it is possible to find studies about SMEs which are related to specific methods of software process improvement. The basic search strings of Table 1 were not intended to search for specific results which contain empirical studies in the software industry (e.g. case studies). So, in line with our research question, we used the inclusion and exclusion criteria to select only the publications which reported case studies (quality threshold).
Science@Direct, on the subject of Computer Science,
Wiley InterScience, on the subject of Computer Science,
IEEE Digital Library,
ACM Digital Library, and
As grey literature, the special report “Proceedings of the First International Research Workshop for Process Improvement in Small Settings” from the SEI was reviewed.
2.3 Studies selection
We propose that the criterion for carrying out the systematic review be based on an iterative and incremental procedure. The term “iterative” indicates the repetition of one or more activities; the term “incremental” indicates that implementation proceeds from an initial subset of the sources to more and more complete subsets, until the whole review is addressed. It is iterative, because the execution (search, information extraction, and visualization of results) of the systematic review is carried out completely on one search source first, and then on the others in turn. It is also incremental in the sense that the document (which is the product) of the systematic review grows and evolves with each iteration until it becomes the definitive one.
Implementation starts from a source, reducing the complexity and scope of each execution.
Each cycle is short and provides feedback for the next cycle.
There is an integration of the results of each cycle into the final review.
This integration can be progressively achieved by successive releases of the review, each release achieving more primary studies.
The increments can be carried out both independently and in parallel.
The complexity of each increment is easier to administer.
The inclusion criterion for what to include as regards studies took as its basis an analysis of the title, the abstract, and the key words from the articles obtained in the search. These served to determine whether or not they were related to SPI and if they focused on, and were applied to, SMEs. Another focus of analysis was how the key words improvement and small were dealt with in the total content of each article, in order to identify whether a given study presents a real case study. The goal was to decide whether or not a given article should be chosen as a relevant study in the context of the systematic review (in other words whether it was a potential candidate for primary study).
A first level, in which the study consists of the application of a process improvement effort that focuses on small and medium-size organizations (there is a case study applied to SMEs, see all studies selected in Appendix B).
A second level, in which the study deals with SPI initiatives for SMEs which have nevertheless never been applied in such companies (where there is no case study applied to SMEs, for instance (Brodman and Johnson 1994) or where it is a theoretical study which is not based on direct/empirical observation).
A third level, in which the study reports on SPI initiatives where some SMEs are involved, but improvement in these companies was not the main objective of those initiatives (see, for example, the study presented in El Emam and Birk 2000).
Distribution of studies by source (January 1996–March 2006)
IEEE Digital Library
ACM Digital Library
In this systematic review potential references within primary studies were not searched for.
2.4 Information extraction
Data extraction format
Data extraction from the study
Practical Software Process Improvement—The IMPACT Project.
13th Australian Software Engineering Conference (ASWEC’01).
Louise Scott, Ross Jeffery, Lucila Carvalho, John D’Ambra, Philip Rutherford.
Objective results of the study
• This paper presents a framework for SPI that is designed to utilise a range of improvement technologies and which supports continuous and highly focused improvement over many projects, thus producing timely, cost-effective and tangible improvements for SMEs.
• The effectiveness of the framework is illustrated in this paper with its application in a small, Sydney-based, web-development company.
IMPACT, ISO 12207
Uses an improvement framework in order to guide the programme SPI.
Guide the improvement programme. Have got freedom in choosing the improvement technology.
Projects documentation, process planning, process estimation and requirements elicitation.
Subjective results of the study
• The framework is cyclic—advocating an “understand—improve—apply—measure” approach which can be applied incrementally across many projects. On each cycle a different focus for improvement is chosen so that each iteration need not require a large expenditure on the part of the company and the improvements are available to all subsequent projects. By implementing many small iterations of the improvement cycle results are achieved quickly (within a few months) and are measurable in the projects in which improvement has been applied. The framework is implemented using existing and proven technologies, frameworks and standards that can be purchased (at a greater or lesser expense depending on the choice of the company) commercially.
• Accordingly, our framework for software process improvement, as shown in Fig. A, differentiates between the project level and the process level. At the project level, many projects are carried out according to good project management practice (e.g. a “Plan–Do–Check–Act” cycle). This is shown as the small cycle in Fig. A which we call the project cycle. At the process level, experience and understanding obtained from many projects are used to understand and improve the generic process model which is then used to guide future projects. This is the large cycle shown in Fig. A which we call the process cycle. Note that these cycles interact closely—the process cycle drives the execution of the projects and the projects drive improvements to the process. The following paragraphs will expand further on each of these cycles and how they influence software process improvement.
During the information extraction phase we assessed the quality of each primary study by means of the attributes: “Bias”, “Internal Validity” and “External Validity”. Each attribute was scored “Yes” or “No” depending upon whether the study explored explicitly the possibility of biases (selection, experimenter or publication) or threats (internal or external validity). From the 45 primary studies none scored “YES” for all these attributes. Seven studies mentioned at least bias issues (primary studies 2, 3, 7, 22, 34, 35, 38—Appendix B). Only two studies mentioned internal and external validity (primary studies 5 and 12). These results may be due to the nature of studies, as they were case studies, and especially the fact that the treatment of internal and external validity threats takes place in the context of formal experiments.
Furthermore, a study design assessment was performed in order to obtain greater insight into the quality of primary studies. The type of design of each study was extracted and classified according to the study design hierarchy for software engineering presented in Kitchenham (2004). With regard to the quality of the publications (from this point of view) included in the systematic review it can be stated that primary studies are observational ones with design type case series, by means of which the formulated improvement proposals were applied to SMEs. From these design type case series, 82% are post-test and 18% are pre-test post-test. Additionally, all studies show a qualitative analysis of the results obtained, whereas just 27% of the studies present a quantitative analysis of some of the results and only 47% of the studies obtained evidence from 2 or more SMEs.
3 Results and discussion
In this systematic review, we have formulated a research question which aims to obtain insight into the current state of practice of SPI efforts in industry. The objective is to use the current state of the art to identify new research activities to carry out in the field of SPI.
A statistical analysis was conducted with the information extracted from the studies, whose purpose was to show the relevant findings of the systematic review that had been carried out. These relevant findings are based on the discussion of significant aspects related to this area of knowledge and include aspects such as size of the companies involved, models used, improvement proposals, processes improved and factors in success.
3.1 Companies involved
The average number of SMEs involved per study is 2.97. This means that in each of the studies an average of three companies is used. We should bear in mind, however, that 53% of the studies involved only one company.
3.2 Models used
Standards used by each study
11, 14, 16, 18, 25, 26, 28, 29, 30, 32, 33, 26, 39, 40, 41, 42, 44
17, 40, 42, 45
12, 18, 26, 39, 40, 42
14, 17, 26, 39, 40, 42, 45
2, 6, 9, 13, 21, 35
3, 4, 20, 22, 38
11, 15, 28, 30, 33, 41
1, 10, 22, 27, 41
8, 19, 23, 31
As can be seen in Fig. 4, the assessment method that is most frequently used in the SMEs is the ISO/IEC 15504, in its 1998 and 2004 versions. Furthermore, the SEI assessments, the CBA-IPI proposals, or less intensive assessments such as the C class or micro-appraisal, follow in order of importance. We could add that the ISO 15504:2004 standard is beginning to be applied on a wide scale, due mainly to the fact that it is a model which is easy to understand, flexible, and fits the needs of SMEs. In some studies, the use of assessment models developed particularly for small companies and based on the ISO 15504 standard is shown, as for example in MARES (Wangenheim et al. 2006) and RAPID (Rout et al. 2000).
With regard to process reference models, that which is most widely used is the CMM (which is expected to be replaced in use by CMMI) of the SEI. In a CMM environment, the SMEs set out to achieve level 2 when an improvement process begins. It is important to underline that from the 122 companies involved in the primary studies, only two medium-size companies of 70 and 150 employees achieved the formal assessment as CMM-SW level 2. We consider that the motivation for these types of companies should therefore be to improve their processes in order to make them more effective and efficient units, rather than to gain the certification itself. Following these processes should be a means to an end and not an improvement goal (Paulk 1998).
Furthermore, from the analysis of the models used, we noticed that 30% of the improvement efforts which used CMM also used ISO 9001. If an ISO 9001-certified organization wishes to continuously improve its processes, the adoption of CMMI would be a good choice, since it provides more detailed practices for process improvement than ISO 9001 (Yoo et al. 2006). However, it is important to note that the compliance with ISO 9001 allows organizations to have broader beneficial effects which are beyond the scope of coverage of the CMM. Although ISO 9001 is a generic standard for quality management (and is therefore not directly concerned with software engineering best practices), we believe that it has been relevant for the software industry as it is more feasible to adopt (in cost and time) that others standards, especially for small companies.
Studies such as Hareton and Terence (2001), Johnson and Brodman (1999) and Saiedian and Carr (1997) show that the models from SEI or ISO are difficult for small and medium organizations to apply because of the complexity of their recommendations and the consequential large investment in terms of time and resources. From the results extracted it can be observed, nevertheless, that these models are the most widely applied in these kinds of companies. It is thus important to consider these models as the reference to develop research works about new models related with the tailoring and adaptation of SPI for SMEs. These new models would allow SMEs to obtain certification in the above-mentioned reference models as they grow in size and resources.
It is significant to underline that in most cases the improvement programmes in small software companies did not lead to the ISO or SEI certifications. The question which arises from this is whether software process improvement standards are suitable for small organizations. From the extracted primary study results and in accordance with the previously exposed paragraphs we consider that these standards per se are not suitable. The reasons why small organizations do not adopt these heavyweight approaches, such as for example CMMI, have been explored in Staples et al. (2007). This reinforces the need to have available strategies for process improvement which are tailored to small companies’ characteristics. These strategies must be aligned with the widely recognised standards in order to enable small companies to establish a solid basis for improvement which will lead them towards certification in these standards in the future. Furthermore, organizations which develop international Software Engineering standards should target the small enterprises separately from the medium enterprises. For instance the WG24 (an ISO/IEC JTC1 SC7 working group) has recently been established to develop international standards profiles in order to help apply them in very small enterprises (fewer than 25 employees).
According to Derniame et al. (1999), the emphasis on processes and on process management provides the main justification for many standardization initiatives of organizations such as SEI or ISO. It is also important to emphasize that process improvement is an integral responsibility of software process management (Florac et al. 1997). Consequently, we believe that it is important to work on how to establish a basic process management infrastructure that is tailored to SMEs and which will support the key principles of process management (to define, to measure, to control and to continually improve the processes). In this sense, and with regard to the field of process measurement, it is important to highlight that from the 45 studies that were analysed, 20 have used the SEI models, 18 have used ISO models and only 2 have used models for the definition of measurement programmes. This tends to support the assumption that the efforts of SMEs are still insufficient in the establishment of formal software measurement processes. Improvements introduced by the SPI are measured by means of informal and non-objective processes based on the employees’ perception and not through formal measurement programmes. This implies the need to dedicate more research effort to software measurement in SMEs, given that this is a core activity in the area of process management, which is a prerequisite to carrying out a suitable control and improvement of the processes.
3.3 Improvement proposals
% of Studies
IP_1. Establish software processes (Use of electronic process guides and experience repository—EPG-ER; Adaptation and use of practices RUP, XP, SCRUM, among others; Informal self-assessment for software developers (web tool))
The software process
IP_2. Prioritize the SPI efforts (by means of DAIIPS method, Software Process Matrix, Express Process Appraisal or Framework of key decisions of business strategy and product development)
The SPI programme
IP_3. Evaluation of a SPI programme (with low-rigour, or to define and use a metrics programme)
IP_4. To guide the SPI efforts (Primer method, Framework IMPACT, MESOPYME method/model, using improvement patterns, following an assessment process, neuronal network)
IP_5. Adaptation and use of SPI models/methods (PSP, TSP, CMM, CMMI, IDEAL, ISO 15504:2004, SPICE, ISO 9001)
IP_6. Others (Definition and use of test framework; Lead process improvement experiment; Use knowledge management for SPI; Acquire ready-to-use technical infrastructure; Improve the relationship and cooperation with clients)
With the aim of providing an initial basis for classification of improvement strategies which synthesize the obtained results, six exclusive categories have been identified (from IP1 to IP6), which are shown in Table 5. These categories were defined by means of an in-depth analysis and subsequent synthesis of commonalities and differences of the SPI proposals extracted from each primary study. First of all, each publication was thoroughly analysed and then a summary of the main ideas and contributions was extracted. The next step was to group the SPI contributions which were related to a more general concept into categories, which resulted in the six exclusive categories. For example, the improvement proposals “to use an improvement framework in order to guide the SPI programme” (see Table 3) and “to follow the concept of action packages in order to carry out the implementation of the SPI in SMEs” (Calvo-Manzano et al. 2002) can be grouped under the category IP4 “to guide improvement efforts”. These categories may evolve in future replications of this systematic review.
The improvement proposals focus on two targets: the software process and the SPI programme. From Table 5, we can observe that what most improvement proposals (71%) have concentrated on is the guidance of an improvement project and the priority of improvement implementation, along with the use of existing improvement models, adjusting them to needs. They have also put great emphasis on the evaluation of the improvements introduced for the SPI program. Alongside this greater bulk of proposals, there are others (18%) which focus on the definition, evaluation, and support of the software processes.
From the analysis of the primary studies extracted, it can be concluded that the improvement approaches can be generated at an organizational or a technical level. It should be emphasized that there are studies (11%) which, in the quest to support actions of process improvement, propose the supplying of technical tools and facilities that support the activities related to the processes. This is termed as technical infrastructure and includes features such as document templates, process models, techniques and tools.
3.4 Improved processes
3.5 Success factors
Initiate the improvement as soon as possible with a simple structure of the SPI model, but before starting the SPI programme ensure that the organisation is sufficiently stable. It is important to have expert consultantship when beginning the SPI programme.
Guide the improvement programme, following a systematic and coherent initiative by means of specific procedures and combining different approaches. This procedure must follow an iterative and incremental approach (prioritize the improvement points defined by the organization) that allows a continuous adoption of improvement practices.
Provide fast return on investment and yield advances to the organization, carrying out improvements that have a maximum effect in as short a time as possible. Furthermore, get optimum value out of resources, stretching them as far as they will go.
Look for external financial aid or join together with other companies so as to both finance and share specialised resources that are involved in improvement.
Involve the greatest possible number of company employees, using personnel training to ensure as few interruptions as possible in the SPI programme.
Minimize resistance to change, through organizational awareness of how the improvement, if put into operation as a response to a real need, will greatly benefit the company.
Track and supervise the SPI programme by means of frequent assessments of processes, in order to evaluate the efficiency of this programme. A good strategy is to carry out rapid assessments of software processes.
Tailor and carry out measurement activities by means of the systematic use of process metrics.
Establish the infrastructure that supports an efficient communication between the different actors involved in the improvement.
The managerial commitment and their strategic plan of the organization are fundamental within an SPI programme. There must be a commitment to an ongoing quest for quality on the part of all the organization’s employees.
Tackle the problem of improvement from the technical perspective.
The review has been performed in the context of the COMPETISOFT project. So, as the group of researchers is small (a researcher and two supervisors), the scope of the revision had to be well defined and therefore, for the first execution of the systematic review procedure the secondary searches were not taken into account.
The observations and conclusions presented are based on the results extracted from 45 primary studies, which is a significant and representative sample of the published works on the topic, in accordance with the quality of the selected sources. Systematic reviews, however, do limit the scope of the population mainly to that of published studies, which can be a relatively low percentage of the overall population. This can limit the power of generalization.
The selected studies include empirical validation based on case studies and all of them presented positive results. Furthermore, because of the nature of primary studies, it was difficult to find quality attributes in the context of case studies. Our assessments about quality attributes are not judgements about whether the primary study had a “good” treatment of these issues (it is limited only to analyse whether the studies considered these issues).
A sensitivity analysis was not applied: (i) with regard to the inclusion of only high quality studies, given the low number and proportion of primary studies which dealt with quality attributes; and (ii) given the heterogeneous nature of primary study results which made the quantitative comparison difficult and the carrying-out of meta-analysis unfeasible (additionally, there are no standardized/universally accepted criteria for reporting results in the field of SPI).
4 Conclusions and future work
Systematic reviews require an effort which is considerably more intensive than that needed in the conventional reviews of literature. The formality with which the systematic review is carried out allows us to validate any results reported in the systematic review which are, in turn, both supported and guaranteed by the review protocol.
In this article a systematic review of the literature on SPI efforts carried out in SMEs has been presented, which has allowed us to obtain a view about SPI in SMEs, using a representative set of related works which was obtained as a result of the systematic review.
From the results obtained in the current review, the first observation from the study that was carried out is that, in the last 5 years, the Software Engineering community has shown an ever-increasing interest in tackling SPI in SMEs. Although at a global level there is a general consensus that SEI and ISO models can be applied only with difficulty in small companies, these same companies do indeed adapt and use these models when undertaking their improvement efforts. Small companies are currently undertaking SPI efforts, but SEI or ISO models do not lead to these companies receiving their certification. The results presented in this paper show that it is indeed very difficult to successfully apply formal SPI programmes which use models such as, for example CMM, to SMEs.
Another relevant fact is that the most frequently used improvement models in the SMEs are CMM (which is expected to be replaced in use by CMMI) as a process reference model, ISO 15504 as a process assessment method and IDEAL as the model for guiding improvement. Nevertheless, there are no studies in which the application of the three models to a small company is reported. It is important to notice that the widespread use of CMM is due to the fact that it was one of the first models on the market and was heavily promoted by the DoD in the USA.
The vast bulk of improvement proposals major on guiding the improvement project, prioritizing SPI implementations, using existing improvement models and fitting them to SMEs needs, as well as on an evaluation of improvement brought about by an SPI programme. Improvements introduced by SPI efforts are, however, measured through informal and non-objective processes, based on the employees’ perception and not through formal measurement processes. There is a smaller percentage of other improvement proposals which have as a focal point the definition, assessment and support of the software process. The focus is also to undertake improvement actions whose fundamental aim is to provide technical infrastructure to support activities related to the processes. Furthermore, as is shown in Fig. 5, improvement efforts aim to improve processes such as project management, documentation, requirement change management, process establishment, configuration management and requirement elicitation. These processes are related to the support, management and improvement of processes.
Using the protocol of this systematic review as a starting point, secondary searches from primary studies and new sources of studies (e.g. Springer or conferences related to SPI) will be reviewed in future iterations, following the suggested procedure for execution of the systematic review. Furthermore, studies which deal with SPI initiatives for SMEs, but which do not report a real case study, will also be analysed. These kinds of studies have been excluded from the present work, as they did not come within its scope.
We would like to thank anonymous reviewers for their useful comments in order to improve this paper, and we acknowledge the assistance of the following projects: COMPETISOFT (506AC287) financed by CYTED, MECENAS (PBI06-0024) granted by the “Junta de Comunidades de Castilla-La Mancha” of Spain and ESFINGE (TIN2006-15175-C05-05) financed by Dirección General de Investigación of the Ministerio de Educación y Ciencia of Spain.