Towards cost-benefit evaluation for continuous software engineering activities

Abstract

Context:

Software companies must become better at delivering software to remain relevant in the market. Continuous integration and delivery practices promise to streamline software deliveries to end-users by implementing an automated software development and delivery pipeline. However, implementing or retrofitting an organization with such a pipeline is a substantial investment, while the reporting on benefits and their relevance in specific contexts/domains are vague.

Aim:

In this study, we explore continuous software engineering practices from an investment-benefit perspective. We identify what benefits can be attained by adopting continuous practices, what the associated investments and risks are, and analyze what parameters determine their relevance.

Method:

We perform a multiple case study to understand state-of-practice, organizational aims, and challenges in adopting continuous software engineering practices. We compare state-of-practice with state-of-the-art to validate the best practices and identify relevant gaps for further investigation.

Results:

We found that companies start the CI/CD adoption by automating and streamlining the internal development process with clear and immediate benefits. However, upgrading customers to continuous deliveries is a major obstacle due to existing agreements and customer push-back. Renegotiating existing agreements comes with a risk of losing customers and disrupting the whole organization.

Conclusions:

We conclude that the benefits of CI/CD are overstated in literature without considering the contextual and domain complexities rendering some benefits infeasible. We identify the need to understand the customer and organizational perspectives further and understand the contextual requirements towards the CI/CD.

1 Introduction

Many software companies aim to benefit from continuous software integration and delivery (CI/CD) practices. In a nutshell, CI/CD practices promise to increase the speed of delivering new features to end-users by automating steps of software delivery, such as testing, integration, and deployment to the business environment (Fitzgerald and Stol 2017; Kumar and Mishra 2016). The idea is that new features and changes are immediately and automatically delivered to end-users without shelving them and waiting for a scheduled release. Rapid delivery of incremental features has the potential to enable fast feedback cycles, continuous experimentation, and to close the gap between software developers and users (Fagerholm et al. 2017; Zhu et al. 2016).

Retrofitting an organization with A CI/CD pipeline is a substantial undertaking. Most existing software engineering practices and architectures need to be adjusted to fit within a continuous pipeline. Other organizational aspects, such as the business model and customer agreements, need to be tailored to support and benefit from continuous software delivery (Fitzgerald and Stol 2014; Neely and Stolt 2013; Chen 2015).

Organizational changes, the introduction of new ways of working and delivering value may disrupt the organization (Giardino et al. 2015). Maintenance of the pipeline, such as tooling, test data, and automated test suites, requires commitment and continuous investments (Sundelin et al. 2020; Garousi and Felderer 2016). However, the risk and investments associated with the adoption of CI/CD are justified only if the benefits are relevant and sufficiently contribute to achieving organizational objectives.

There is a substantial body of knowledge around how to implement elements of a CI/CD pipeline and overcome the associated challenges; see, for example, Olsson et al. (Olsson et al. 2012), Rodriguez et al. (Rodríguez et al. 2017), Fitzgerald et al. (Fitzgerald and Stol 2017) and Laukkanen et al. (Laukkanen et al. 2017). However, very few studies address the holistic cost/benefit aspect of CI/CD. The understanding of costs and benefits is required to make informed decisions of what parts of CI/CD are relevant for a given product and organization. Especially, if the investments are substantial and changes carry organizational risks.

Naturally, organizations aim to maximize the benefits of their investments. However, we observe that the adoption of CI/CD practices is often driven by hype and promises of huge, however vague, benefits. For example, CI/CD are ought to enable more frequent software releases and flexible business models (Humble and Kim 2018). However, how to assess the relevance of such benefits for a given company, and gauge whether the investments can be justified with the expected gains is not known. Without a detailed analysis of required investments, the relevance of the benefits, and any downsides, organizations may waste resources on implementing irrelevant practices, too expensive to maintain or even harmful to a successful business.

Organizations may also miss capitalizing on the benefits of CI/CD while incurring the cost. This is due to the potentially asymmetrical nature of costs and benefits. The costs could be incurred at one stage of the software delivery pipeline with a small perceived immediate benefit. However, the investment’s actual benefits could be reaped at a later stage, only if such a later stage is implemented and aligned correctly (Laukkanen et al. 2015). The same is true upstream. For example, achieving regular and continuous customer feedback due to the pipeline might significantly benefit product management and product planning. However, if that part of the organization is not prepared to utilize said feedback as a part of their work, significant parts of the benefits are not realized.

The term CI/CD largely refers to automation aspects of software engineering. However, Fitzgerald et al. (Fitzgerald and Stol 2017) points out that there are other continuous activities, such as planning, support from the organization, business models and customers, that enable and maximize benefits of the automation efforts. Thus, we explore continuous software engineering from a holistic cost-benefit perspective encompassing all connected aspects. We map state-of-practice in two companies to state-of-the-art to identify aims, expected benefits, investments and prerequisites to adopting specific CI/CD practices. We further discuss the relevance of the potential benefits to our studied cases and highlight areas with insufficient support from state-of-the-art.

The rest of this paper is structured as follows. Section 2 presents background and related work, Section 3 outlines the research methodology, Sections 4-5 presents and interprets our findings, Section 6 concludes the paper.

2 Background and Related Work

2.1 Agile, Lean and Continuous Software Engineering

Agile software engineering principles emphasize working software over extensive planning and documentation, rapid response to change, and collaboration versus contract negotiation (Fowler et al. 2001). These principles have inspired a large number of agile software engineering practices (Jalali and Wohlin 2012; Martin 2002; Sidky et al. 2007). Most importantly, agile practices aim to deliver value in small increments, facilitate collaboration and anticipate changes in customer needs and the environment (Hazzan and Dubinsky 2009; Alahyari et al. 2017).

Lean software engineering is an adaptation of lean manufacturing principles in the software domain (Poppendieck and Poppendieck 2003). Lean principles are often applied in conjunction with agile practices (Rodríguez et al. 2012; Alahyari et al. 2019). Both lean and agile emphasize rapidly delivering customer value while minimizing activities that do not deliver value.

Continuous software engineering is a paradigm to deliver new software features to end-users in small increments and as rapidly as possible. The speed is achieved by combining lean and agile principles with extensive automation and removal of organizational silos (Chen 2015).

In continuous software engineering, software travels through a series of interconnected and largely automated steps to deliver the latest software changes to end-users with minimal delay. The automated steps handle testing, integration, delivery of software (Humble and Kim 2018). Once software is delivered and used by end-users, telemetry is relayed back to the software vendor to support further product decisions.

Fitzgerald et al. (Fitzgerald and Stol 2017) identifies the following components of a continuous software delivery pipeline:

• 1. Continuous product planning considers various inputs (customer feedback, market data, etc.) and prepares plans on leveraging software engineering to attain organizational objectives (Lin 2018; Provost and Fawcett 2013)

• 2. Continuous software integration comprises continuous development, configuration management, testing, integration, and other activities to produce working software (Meyer 2014; Felidré et al. 2019; Hilton et al. 2017).

• 3. Continuous deployment making the latest software features available for delivery to end-users (Zhu et al. 2016; Senapathi et al. 2018).

• 4. Continuous delivery refers to the ability to enable end-user access to the latest features, i.e. updating production software, at any time and with minimal delay (Dubey and Wagle 2007; Chen 2017; Mäkinen et al. 2016)

• 5. Continuous use & trust arising from less disruptive releases and improved value delivery (Gefen et al. 2003; Susarla et al. 2009).

• 6. Continuous feedback collecting customer feedback to support further product planning (Guzman et al. 2017; Lin 2018; Provost and Fawcett 2013; Fabijan et al. 2017)

• 7. Continuous improvement measures the performance of software delivery and fine-tuning the pipeline. Metrics such as cycle-time, batch size, mean-time-to-recovery, and throughput are proposed by Humble et al. (Humble and Kim 2018) to monitor the pipeline.

There are many studies focusing on the implementation, challenges, and potential solutions of individual components of CI/CD, especially the continuous integration (CI). However, we identify lack of a holistic discussion on the relevance and interrelations of all continuous practices.

For instance Hilton et al. (Hilton et al. 2016) analyzes continuous integration (CI) in open-source projects. Their findings show that the use of CI correlates with more frequent releases and popularity of open-source projects among other benefits. The main obstacles in adopting CI are lack of familiarity with CI, lack of test automation, and a slow rate of contributions to the project. Elazhary et al. (Elazhary et al. 2021) analyzes benefits and trade offs of CI practices. One of their main findings is that due to differences in implementations, contexts, and perceptions, CI cannot be studied as one practice. Rather, a more fine-grained view is needed to analyze CI.

To our best knowledge, there are no similar studies to e.g., Hilton et al. (Hilton et al. 2016) and Elazhary et al. (Elazhary et al. 2021), analyzing the suitability of continuous practices throughout the whole software life-cycle, as suggested by Fitzgerald et al. (Fitzgerald and Stol 2017). Furthermore, most existing work analyzes state-of-practice in organizations who have already adopted continuous practices to some extent. This approach creates a sampling bias and excludes organizations who have not yet overcome the adoption threshold from analysis.

Our study contributes to state-of-the-art twofold. Firstly, we analyze continuous practices, associated investments, benefits, and contextual factors from a holistic perspective. We consider the whole software life-cycle. That is, from the inception of a feature in planning stage to analyzing customer feedback on the feature and using it in fine-tuning the feature as suggested by Fitzgerald et al. (Fitzgerald and Stol 2017).

Secondly, we analyze CI/CD in two organizations in telecom domain who have not yet achieved operational level on any of the continuous practices. This perspective allow us to analyze contexts where CI/CD adoption is challenging and requires most support.

2.1.1 Cost-Benefit Analysis

The cost-benefit analysis is a method to evaluate decisions in terms of their consequences. That is, to what extent the investment (cost) in a project is justified by the gains (benefits) of the project (Drèze and Stern 1987), where the ”project” is the change required to realize the CI/CD environment. The cost-benefit analysis method is widely applicable and could be tailored to fit a variety of scenarios. The method comprises the following general steps (Sassone and Schaffer 1978):

• 1. Identify stakeholders and their goals concerning the project being evaluated.

• 2. List alternatives. It could be a decision between maintaining the current or moving to a new state or deciding between any number of alternatives.

• 3. Define metrics to use in the evaluation. The metrics qualify the stakeholder goals from Step #1.

• 4. Determine the costs and benefits associated with the project.

• 5. Establish a timeline of when costs are incurred and benefits reaped.

• 6. Express the costs and benefits in similar units. Similar units are required to perform a direct comparison between investments and benefits.

• 7. Calculate the net present value and ratio.

• 8. Perform sensitivity analysis. Determine to what extent adjustments in the inputs (investments) affect the consequences (benefits).

• 9. Make decisions.

To support the application of the cost-benefit analysis for evaluating the adoption of continuous software delivery practices, there is a need for an improved understanding of:

• 1. Common stakeholders of a CI/CD pipeline and detailing of their respective goals

• 2. A fine-grained understanding of investments (costs) and benefits (estimated) associated with CI/CD and its components

• 3. Relevant metrics to gauge the progression towards said goals

• 4. A methodology to quantify and compare different benefits and costs

• 5. A methodology to determine a configuration of continuous practices given the context of the organization

King and Schrems (King and Schrems 1978) propose using cost-benefit analysis to evaluate decisions regarding software development and operation. They identify possible types of project benefits, such as:

• Cost reduction or avoidance

• Error reduction

• Increased flexibility

• Increased speed of activity

• Improvement in management planning or control

Furthermore, they identify the following types of project costs (investments):

• Procurement costs of acquiring tools, equipment, etc.

• Start-up costs incurred due to starting the project, e.g., adopting tools and practices, training, and disruption to the organization.

• Project-related costs such as personnel, overhead, data collection, and software modifications.

• Ongoing costs, for example, maintenance of software and hardware.

The use of cost-benefit analysis to evaluate decisions in various areas of software engineering, such as adoption of cloud computing (Maresova et al. 2017; Chandra and Borah 2012), software architecture (Carriere et al. 2010), and requirements (Letier et al. 2014), is not new. Common challenges in applying cost-benefit analysis include identifying the right metrics, expressing all costs and benefits in common units, and identifying all plausible alternatives for comparison (King and Schrems 1978). Our work aims to identify possible benefits, costs, and requirements under which an organization can realize the benefits. Such a map is the first step in aiding management and other pertinent organizational units to invest in implementing CI/CD and maximizing its operational benefits.

3 Research Methodology

3.1 Aims of the Study

This study aims to explore CI/CD from a cost (investment) and benefits perspective. Specifically, we are interested in what benefits and associated prerequisites to these benefits are linked to continuous practices. We also try to ascertain which benefits (as reported in the state-of-the-art) are considered relevant in state-of-practice (industry).

3.2 Research Questions

To attain our objective, we set forth the following research questions:

RQ1: What are the key steps of a CI/CD pipeline?

Rationale: CI/CD is an umbrella term to describe a series of mostly automated and interconnected practices. We aim to break down the concept into its constituents to support our further and more detailed analysis. We go beyond (sprint-based) engineering and include product management/planning, realization, but also delivery/commissioning and maintenance/evolution of the software-intensive product and service development.

RQ2: What potential benefits and potential investments (costs) are inherent to the CI/CD adoption?

Rationale: We aim to understand what motivates organizations to adopt CI/CD practices, what benefits motivate this, and what investments are required. We also analyze the differences between benefits triggering the adoption of CI/CD and the benefits with the most relevance

RQ3: What is needed to support the CI/CD cost/benefit analysis in the industry?

Rationale: We aim to identify opportunities for further research into the benefit/consequence analysis of continuous software engineering practices.

RQ4: What are the gaps between state-of-the-art and state-of-practice in relation to CI/CD?

Rationale: We aim to identify underdeveloped areas of CI/CD to aid further research into the area.

3.3 Research Method

We answer our research questions by conducting two industrial case studies supported by a structured literature review (not to be confused with a systematic literature review). With the case studies, we explore practitioners’ needs and aims towards CI/CD. From state-of-the-art, we develop a conceptual model of continuous software delivery and use it to connect state-of-the-art to state-of-the-practice.

Both the structured review and case studies are conduced in parallel and support each other. In Fig. 1 we show an overview of our research methodology.

Fig. 1

Overview of the research method. Alignment between case studies and the structured review

3.3.1 Industrial Case Studies

We analyze the adoption of CI/CD at two companies, Ericsson AB and Telia Company AB, using in-depth case studies and following the case study research methodology proposed by Runeson (Runeson et al. 2012). The object of our study is the continuous software engineering process, related practices, relevant organizational aims, and challenges from the benefit-consequence (cost) perspective. We limit the scope of our inquiry to one specific product in each company.

The data collection took place between September 2020 and April 2021. Each organization was involved in capturing the broad view from multiple perspectives affected by and instrumental to the area, including developers, project managers, product managers, line-manager, test engineers, and co-workers representing the operations perspectives.

We structured the interviews and workshops to familiarize ourselves with each case and their software delivery process specifics, see Step CS1 in Fig. 2. We perform inventory of their current process and continuous practices. The purpose of the inventory is to understand what practices (continuous or not) are currently applied in each case (RQ1), why, what benefits could be realized by adopting continuous practices, and what are the associated investments to realize said benefits (RQ2). Prior to the company engagements we designed a spreadsheet to keep track of the gathered data, see the structure of Spreadsheet A in Table 1.

Fig. 2

The conceptual model of state-of-the-art CI/CD pipeline

Table 1 The structure of Spreadsheet A to support inventory of continuous practices at each company

We further aim to capture broader organizational goals to gauge how continuous software engineering can be useful and what parts of the pipeline are relevant to attain these goals. Looking at the broader organizational goals we identify focus areas to steer our further investigation.

We aim to complement our discussions with with document reviews, for instance process overviews, presentations outlining organizational aims, and strategies. Such materials will serve as input for steering discussions in the workshops.

As we discover new angles and perspectives, we aim to schedule additional data collection activities with additional stakeholders to capture these new perspectives. As we gain insights from the case studies, we update our conceptual model and analyze the gap/overlap between state-of-the-art and state-of-practice. With this analysis, we identified overlapping parts, gaps, and areas for further discussion.

We analyzed both cases and looked at state-of-the-art in parallel. Thus, findings from one case could be immediately cross-checked with the other case and state-of-the-art, see steps CS4–6 and R4–5. Once our analysis reached thematic saturation we finalized our reports, and presented the results to the cases for additional input and feedback.

Once our case studies reached theoretical saturation and no new concepts emerged we finalized our conceptual model. The conceptual model, see Fig. 2, is a emerged from the case studies while maintaining connection to state-of-the-art. The final iteration of the model was presented to both studied cases for additional input and feedback.

3.3.2 Connection to State-of-the-Art

To design and support the case studies, we refer to state-of-the-art. Our aim is to use state-of-the-art to establish a baseline perspective of CI/CD to support our interactions with practitioners. Specifically, we are interested to what extent benefits and investments of continuous practices from literature are relevant for the studied cases.

Our approach should not be confused with a systematic literature review or mapping as defined by e.g. Kitchenham (Kitchenham 2004) or Pettersen (Petersen et al. 2008). Rather we build upon existing secondary studies and look for papers linked in these studies to explore each step of CI/CD.

The initial scope of the review was defined by using the CI/CD research roadmap (Fitzgerald and Stol 2017). The roadmap by Fitzgerald et al. (Fitzgerald and Stol 2017) was selected as it provides the a comprehensive overview of CI/CD concepts with relevant references. The roadmap was complemented with results from a systematic literature review (SLR) (Shahin et al. 2017). To gain a more in-depth perspective, we performed a backward snowball sampling iterations, i.e. followed the references, from the research roadmap and the SLR to find the primary studies discussing pertinent concepts, see steps R1–2 in Fig. 1.

We performed narrative analysis of the discovered papers to gain an overview of pertinent themes, associated issues and concerns relevant to our case studies (Huang et al. 2018). With the analysis we extracted practices, benefits, associated investments, challenges, and preconditions for applying the practices and realizing their benefits. Extracted data were organized in a Spreadsheet B, With this input we formulated the preliminary conceptual model, see R3, in Fig. 1, and prefilled the inventory spreadsheets with potentially relevant insights for discussion in the workshops. Different slices of the Spreadsheets A and B are presented in tables in the results section.

As the study progressed and more input from the cases was collected, see Steps CS3–5, in Fig. 1, we performed additional literature searches and thematic analysis, to support the case studies and evolve the conceptual model further, see Steps R4–5. Simultaneously, we expanded the scope of the inventory to reflect new findings from literature.

3.4 Threats to Validity

Construct Validity

captures to what extent the operational measures captures the intended phenomenon (Runeson et al. 2012). A potential threat in this category stems from different understanding and terminology between researchers and industry participants. To minimize this threat, we start or interactions with industry participants with a presentation of the conceptual model. We further ask the participants to point out parts of the model known to them and single out new parts. Through such discussion, we establish a common view on the studied phenomena.

Another treat arises from our review methodology. Our aim was to gain a broad and good-enough support for conducting the case studies and to identify discrepancies between state-of-the-art and state-of-practice. There could be relevant studies that study overlooked. To mitigate this threat we divided our review per CI/CD step to more precisely target pertinent literature, and continuously compared state-of-the-art and state-of-practice to gain as broad coverage as possible.

State-of-the-art presents continuous practices spanning the entire life-cycle of a feature, from the inception of a feature idea, development, delivery to customers, use, collection of feedback, and use of feedback to steer the next iteration. However, our studied cases offer primarily focus on development and delivery steps. Therefore, there is a threat that the other steps of continuous engineering are not sufficiently covered.

Our studied cases have implemented or attempted to implement on some of the continuous practices. Thus, their input on the other practices is based on their expert opinion, not first-hand experience. This introduces a threat that they may have overlooked or misinterpreted some aspects of such practices. To counter this threat, we compare and point out differences between state-of-practice and state-of-the-art.

Internal Validity

concerns the causal relationships and potential confounding factors to the explored phenomenon (Runeson et al. 2012). In this paper, we do not attempt to establish causal relationships. Thus, this category of threats is not relevant.

External Validity

concerns the extent of generalizability of presented results (Runeson et al. 2012). The empirical part of our study considers two cases from the telecom domain. Thus, the generalizability of our empirical findings may be limited to similar domains and types of companies.

In reviewing state-of-the-art, we consider all sources irrespective of domain, product, and market type. Thus, our review represents a broad view of the phenomenon. We devise the proposed conceptual model from earlier systematic literature reviews. That said, the mapping between state-of-the-art and state-of-practice is limited to our two cases belonging to telecomunications domain and relatively mature companies. State-of-the-art may have additional limitations that we did not discover due to the nature of our studied cases. For these reasons, the conceptual model could be skewed towards the studied cases.

Reliability

examines to what extent the data and the analysis are independent of specific researchers (Runeson et al. 2012). The first author conducted the workshops, interviews, and other data collection activities. During the data collection activities, we used our conceptual model to guide and let participants drive the discussion. Thus, the focus areas highlighted challenges, and other findings emerged independently from researchers. After data collection activities, the collected data was reviewed and discussed by both authors. These discussions helped to shape ideas and focus areas for upcoming data collection activities. During continued interactions with practitioners, we frequently discussed our intermediate findings to gain additional insights and validation. Stakeholders reviewed the final draft of the paper from both companies to ensure that the presented findings are accurate.

Another concern to the reliability of our study is replicability of the results.

4 Results

To understand drivers behind the adoption of CI/CD practices, we study state-of-practice in two companies and map our findings with state-of-the-art. Using this mapping, we highlight parts that overlap, are missing in state-of-the-art, but also items overlooked by practitioners in state-of-the-practice. We differentiate between 5 types of mapping points, see Table 2.

Table 2 Types of mapping points between state-of-the-art and state-of-practice

We structure our results in two parts. In the first part, we present two industrial cases highlighting their contexts, wants, and needs in relation to CI/CD. In the second part, we present state-of-the-art perspecitve, connect it to the state of practice to identify gaps, discrepancies, and relevant results to support the adoption of CI/CD in practice.

4.1 Case I: Ericsson

4.1.1 Research Context

Ericsson AB is a large telecommunications hardware and software provider based in Sweden. The company offers an extensive portfolio of software-intensive products. They combine their offerings with 3rd party products to create customer-specific solutions. Customer solutions are highly customized and delivered to mobile network service providers globally.

In this study, we focus on a product aimed to support mobile telecommunications operators’ business functions (Product A). It is a large and complex constellation of different components. Individual software components have a varying degree of continuous integration and deployment capabilities. However, the challenge lies in the solution-level integration and testing. Differences between components, individual component delivery methods, customer-specific customizations, dependencies on third-party components, and domain regulations make final software integration complex, slow, and manual effort-intensive.

Our primary contact in Ericsson was a service delivery organization. The organization is responsible for orchestrating Ericsson’s different activities to attain efficient software deliveries to customers and align the delivery process with their needs. Recently, they have started an initiative to create a strategy for using CI/CD to streamline the software delivery process, focusing on implementing a universal pipeline for delivering the Product A.

4.1.2 Data Collection

We organized the data collection activities in Ericsson between September 2020 and March 2021, see Table 3. In addition to meetings and workshops, Ericsson provided access to a large number of slide decks containing information about their ongoing work on streamlining software delivery. We used these resources to prefill our inventory spreadsheet and steer discussions in the meetings.

Table 3 Data collection activities with Ericsson

Due to the size of the product and large number of stakeholders involved, the inventory spreadsheet was filled in incrementally. Stakeholders were able to provide information only in their respective areas of responsibility. In one of the final workshops we gathered a larger group of stakeholders to discuss the overall picture. We also jointly developed a mind map capturing the organizational goals, links to specific engineering challenges to be solved, pertinent continuous practices, and obstacles on implementing the practices.

We summarizing the insights from Ericsson illustrating their case in the following subsections. We discuss the mapping between our empirical findings and state-of-the-art in Section 4.4.

4.1.3 Steps of the Software Delivery

The software delivery in Ericsson is a multi-stage process. It can be summarized as follows:

• 1. Individual components are developed using a varying degree of continuous development practices. The latest components are placed in a repository for further integration.

• 2. Service delivery organization combines different components into customer solutions. Frequently, customer solutions contain bespoke customizations.

• 3. Customer solutions are deployed in a staging environment where both software vendors and customers verify the software.

• 4. Once accepted, a solution is taken to customers’ premises for further validation, configuration, and integration with customers’ systems. It may take several months and up to a year until an accepted solution goes live.

• 5. There is a predefined update schedule determining when customers should be prepared for upgrading their solution. The interval between upgrades spans multiple years.

More often than not, transitions between steps of the software delivery are a pull rather than push operation. The reasons include ensuring control and stability of staging and operational environments and service level agreements between the vendor and customers. For similar reasons, verification of releases, especially regarding compliance and non-functional qualities, is performed manually and as a separate step before deployment.

4.1.4 Aim I - Rapid Delivery of New Technologies

Ericsson aims to remain on top of the market with the rapid delivery of innovative offerings. Emerging technologies such as 5G will require software vendors to react more rapidly to new market needs and provide innovative solutions on short notice. The current release pace is not sufficient to address this objective. Specifically, the preparation (by vendor) and adoption (by customer) of releases are overly time-consuming, complex, and expensive. As one interviewee reflected:

“With 5G we must be much faster in rolling out new services. The technology is going to be much more dynamic. We can’t wait 6 months to deliver something and then 6 more months to get feedback.”

To address this objective, the study participants identify two areas of improvement. Firstly, rapid software delivery depends on streamlined development supported by automation—secondly, efficient solution level integration and deployment to staging environments.

Continuous development is practiced to a significant extent. Test, build, and low-level integration is largely automated already. However, due to compliance requirements and legacy processes, some verification steps are performed manually and create a bottleneck at the end of the development phase.

Solution level integration, development, and staging are implemented to deliver highly customized solutions and to jointly with a customer to verify that a release is stable and reliable for adoption. Much time is spent ensuring the reliability of the release and waiting for the proper launch window to upgrade customers’ systems.

The customer systems are often piecemeal solutions consisting of various components from various vendors and in-house developments. Thus, upgrading part of a system may require upgrades and additional developments in 3rd party components. An important part of Ericsson’s offering is to provide solutions that fit within existing customer systems. There is an ongoing initiative to explore to what extent the solution automation can be streamlined and automated. As an interviewee described customer solutions:

“Customers usually develop their systems over time and look for what is more economical for them. A typical solution contains some of our components, components from other vendors, and some of their own software. Ericsson puts in a lot of effort to make sure our components are compatible with whatever customers have built.”

Ericsson needs to implement and streamline their continuous development, deployment, and, potentially delivery, practices to fulfill this aim. However, state-of-the-art offers little support for practicing CI/CD in complex regulated environments; see rows 1–3 in Table 4.

Table 4 Comparison of lessons from state-of-the-art and state-of-practice (Case I)

4.1.5 Aim II - Universal Pipeline to Deliver Software

There is an aim to simplify and unify the software delivery process for different offerings and customers. The current software delivery process is tailored to the specifics of individual offerings and the needs of specific customers. Thus, there is little standardization and, consequently, much room for optimization.

Interviewees identify that the product architecture, for example, component scopes and interfaces, are not optimal for continuous solution level integration. The architectural shortcomings increase solution complexity, thus making efficient integration and delivery difficult. As described by a workshop participant:

“We have a “red team” of our top engineers. Initially it was planned only for incident response. However, they are involved all the time whenever we are preparing a release.”

To remedy this, Ericsson needs to evolve their architectures with CI/CD in mind. Literature suggests that cloud-native components with few interdependencies perform best. However, to what extent investments in evolving existing products to fit well with CI/CD can be justified is unknown.

4.1.6 Aim III - New Business Models

More frequent software deliveries and access to feedback could enable closer cooperation between Ericsson and customers. Thus, enabling new collaborative business models.

Our study participants reflect that this is a very vague aim that needs more details to judge whether it is relevant in the telecom domain. Further discussion revealed that developing new business models must support the existing offerings until all customers are upgraded to the new models. Some customers may never accept continuous deliveries. This creates a potential situation when the organization offers the same product with continuous deliveries and plan-driven software releases. Quoting one participant:

“Sure, it sounds exciting! However, telecom domain has been very slow in adopting innovations in business models. Largely, because we do what operators want and some operators do not really want any innovation.”

Support for parallel business models and software delivery methods adds new requirements and complexity throughout the organization. For example, software must be developed with support for both the old and the new delivery method. Both delivery methods need to be synced to ensure consistency in terms of product features and quality. Customer support must be prepared to assist in both scenarios.

State-of-the-art assumes that the product is always delivered continuously (or release-based) and offers little support for transitioning from one model to another, see row 6, in Table 4.

4.1.7 Aim VI - Data-Driven Decision Making

Ericsson aims to benefit from data-driven techniques in improving both internal processes and customer offerings. To attain this objective, Ericsson needs to implement metrics to measure the software delivery process and establish means of getting access to product telemetry.

The study participants reveal that there have been attempts to gain access to product telemetry and usage data. However, customers are unwilling to share any data to protect their business secrets and the privacy of end-users. As one participant reflected:

“We have had endless discussions to get access to some telemetry. However, our customers see that as a risk rather than an opportunity.”

Ericsson has only partial control over the software delivery pipeline. The software vendor controls the pipeline internally until the latest release is ready for delivery. However, the vendor has limited control of whether customers upgrade to the latest version and no control or access to software once it is operational. One interviewee shared an old but relevant anecdote:

“Once in the ninethies we delivered a new solution with both hardware and software to an operator. It cost them a lot. Our sales teams tried to follow up and learn how satisfied the customer is. However, no useful information came back. Finally, six months later we sent a representative to investigate. It turned out that the operator had not even unpacked the boxes yet.”

Recently with the adoption of cloud-based solutions, the situation has improved. However, the challenge of gathering data from customers is still relevant.

4.1.8 What is Needed to Support Retrofitting Product Product A with CI/CD Pipeline?

Analysis of Case I shows that retrofitting a product with a CI/CD pipeline is a substantial undertaking and introduces many business risks in addition to technical challenges. For example, from customers’ perspective switching to a new product release model and renegotiating service agreement may trigger consideration of alternative offerings by competitors.

Comparing lessons from state-of-the-art and Case I, see Table 4, we identify the following areas for support.

Component Granularity and Flexibility

Current granularity of components is not well suited to support different configurations of customer solutions. Thus, there is additional bespoke development at each release to tailor the solution to customer needs, see rows 1-4 in Table 4.

Component boundaries and limits of flexibility need to be revised to streamline the release process and keep bespoke development at minimum.

Support for Different Parallel Delivery Models

Not all customers could be ready for changes in the product and its delivery method. Customers have built their own and tightly coupled solutions on top of Ericsson’s offerings. Thus, any changes in architecture, components, interfaces, etc., will have a substantial impact, see rows 1–2,4–6 in Table 4.

Currently, customers rely on a predefined release pace with ample time to prepare other systems and their business processes for integration with the new release. Thus, different customers may require different release paces and extent of backward compatibility. Consequently, Ericsson must be prepared to roll out any changes in components and release pace softly while maintaining full support for customers who do not wish to change right away.

Ericsson needs to balance investments, risks, and potential benefits to decide whether it is worth the investment to change the product architecture and upgrade current customers to the improved software delivery model. We summarize the decision in Table 5.

Table 5 CI/CD investment, benefit, opportunity and risk conundrum in Case I

Our interviewees state that the organization has little appetite for the risk of losing existing customers. Thus, the focus is set to adopt CI/CD with minimal disruption to customers. However, the objectives to change the software delivery model without affecting customers are contradictory. More understanding of the customers’ perspective and willingness to switch to continuous software deliveries is needed.

4.2 Case 2: Telia Company

4.2.1 Research Context

Telia Company AB is a telecommunications services provider in Sweden. The company provides telecommunications services to private and business customers. To serve its customers, the company integrates in-house built software with off-the-shelf products.

The current release pace is slow due to domain complexities and dependencies on other systems with long release cycles. However, there is an ongoing initiative to improve internal efficiency and speed up software releases. The initiative is not specific to CI/CD. However, continuous software delivery is considered to be one of the candidate solutions.

The company had selected one of their products (from now on, Product B) as a pilot case for adopting continuous engineering practices. The product is a software layer between other systems supporting an exchange of text messages. The product is stable, mature, and most developments concern maintenance updates and occasional bug fixes. The product is not exposed to customers directly. Instead, it enables features for other products with user interfaces.

4.2.2 Data Collection

We conducted several workshops with the product team to understand their objectives, motivation, constraints, and challenges in adopting CI/CD practices. The product manager, two senior developers, delivery manager, and operations specialist where the core participants. The data collection took place between September, 2020 and April 2021, see Table 6.

Table 6 Data collection activities with Telia

4.2.3 Steps of Software Delivery

Product B is relatively small and developed by a loose team of few developers. Other stakeholders include an operations team responsible for deploying and ensuring the smooth operation of the product. The steps of delivering Product B are:

• 1. Developers receive issue tickets from the operations team and implement necessary changes. The flow of tickets is slow, and developers prepare two releases a year.

• 2. Finished software is complemented with extensive instructions for deployment and operation and forgo several rounds of testing.

• 3. Once a release is ready, operations teams take over and deliver the release to customers with support from the development team.

4.2.4 Aim I - Simplify the Development and Build Process

Due to strong coupling with other systems, Product A requires a complex environment for development and testing. Setting up the local environment and ensuring the correct configuration for development and testing is a complicated task. The team aims to benefit from containerization, build scripts, and similar practices to alleviate software dependency management issues. As one participant summarized the challenge:

“It used to take about two weeks to set up a development environment from scratch. The product has many intricate dependencies that are documented nowhere. Most of the time is spend on troubleshooting compatibility issues. With Docker containers we do not have this issue anymore.”

The slow-release pace of two releases per year has created an issue of knowledge preservation. Over long periods of inactivity on Product B, developers are assigned to other tasks or may have left the company. Every small change requires context switching and re-learning the specifics of the product. As a consequence, developers spend substantial time reading documentation and setting up individual development environments.

4.2.5 Aim II - Reduce the Complexity of Software Upgrades

Currently, every release of Product B is complemented with extensive documentation. Preparing this documentation is a substantial undertaking. The exact audience and value of the documentation are unclear to the development team. However, some documentation (e.g., installation instructions, test cases) is required as software artifacts are passed from development to operations. As described by one interviewee:

“Deployment of the product is not straightforward and operations often need our support to get things going even with all the documentation we provide. I think with automated scripts and containers we can make deliveries easier and leave less room for issues.”

The software vendor aims to streamline software delivery process by tearing down organizational silos and closer collaboration between development and operations. Such a move should reduce the need for extensive documentation and streamline the development-operations process as there would be one team responsible for the whole process.

Current customer agreements determine the software delivery process and do not support frequent releases. Furthermore, customers are not eager to receive frequent software deliveries. They mostly contain maintenance updates with little perceived value and increase the risk of system outages and other issues. The software vendor aims to simplify the release adoption process from the customers’ viewpoint. One participant reflected on this:

“The upgrades does not provide much new features, thus there is little incentive to upgrade right away. We have to send a notice weeks in advance to let customers prepare for an upgrade. We know very little what it takes for them to upgrade.”

4.2.6 Aim III - Improve Software Quality

Currently, Product B forgo several rounds of testing in different test environments before a release. Although some tests are automated, there is considerable potential to benefit from automated testing and verification practices.

Some tests are difficult to automate because the tests are complex, expensive to run, and prone to distrust in automated testing practices. These tests remain as a bottleneck at the end of the otherwise automated integration and verification pipeline.

Due to the high turnaround of developers, the product had eroded and accumulated some technical debt. To support the refactoring effort, the team wishes to benefit from automated and continuous verification practices.

The software vendor aims to benefit from continuous integration and deployment practices to remove bottlenecks of manual testing, simplify the release process, and improve the overall quality of the product. As stated by one interviewee:

“We have started automating some tests and it shows results. However, we still have some pretty heavy manual tests at the end. It would be good to automate them as well. However, we feel that sometimes there is distrust in automated tests and stakeholders are more confident with manual testing.”

4.3 What is Needed to Support Retrofitting Product B with a CI/CD Pipeline?

Currently, there are no inherent obstacles to retrofitting Product B with a continuous integration pipeline. However, the team reflects that by adding new practices and ways of working, they need to revise and remove any obsolete activities. For example, multiple testing rounds on different environments are no longer relevant if the product is containerized. Similarly, deployment scripts can replace installation instructions.

Identification of obsolete steps in development, integration, and deployment is essential to avoid unnecessary investments in automation. However, changes in the software delivery process need to be communicated and explained to all stakeholders.

Our interviewees admitted that they mainly consider the engineering perspective. The interests of other stakeholders, namely customers and adjacent products, have not been considered. Further work is needed to understand software deliveries from customers perspective and to what extent continuous software deliveries reduces the effort to adopt a new software release.

Comparing reflections from participants with state-of-the-art, see Table 7, we observe that practice agrees with literature. Specifically, continuous practices reduces the complexity of software engineering and removes tedious setup and configuration steps, and reduces the need for detailed documentation.

Table 7 Comparison of lessons from state-of-the-art and state-of-practice (Case II)

Continuing adoption of CI/CD, needs to explore the interests of other stakeholders. For example, exploring the potential of aligning software delivery pipelines across the whole organization. Thus, creating a coherent software delivery pipeline that can be further extended to customers organizations. We summarize the investment-benefit conundrum in Table 8.

Table 8 CI/CD investment, benefit, opportunity and risk conundrum in Case II

The CI/CD initiative in the Product B was intended to pilot continuous software delivery within the organization to gauge its further potential. Thus, attaining more than “just enough” continuous capabilities could be a worthy investment to gain a complete picture of the potential benefits, investments and pitfalls of CI/CD in their specific context.

4.4 Connection to State-of-the-Art

A substantial amount of research addresses individual building blocks of a CI/CD pipeline, e.g., test automation (Wiklund et al. 2017; Raulamo-Jurvanen et al. 2017; Kasurinen et al. 2010). However, a big picture perspective is needed to identify all relevant components of a pipeline (RQ1) and understand how individual components of a pipeline fit together to analyze their benefits (RQ2) (Frank 2000).

Inspired by Fitzgerald et al. (Fitzgerald and Stol 2017), we develop a conceptual model visualizing state-of-the-art CI/CD pipeline, see Fig. 2. Importantly, the model is based on state-of- the-art in the area. We use it as an overview and tool in our industry workshops to aid discussions of costs, benefits, relevance, and dependencies between different parts of the pipeline.

In the figure, we denote the main pipeline steps with rectangular blocks. With dashed boxes, we illustrate the key benefits and investments associated with each step. The flow through the pipeline is denoted with arrows. Components of the pipeline are organized by the four main levels of stakeholders in a CI/CD pipeline:

• Level I: Engineering organization produces software. The primary objective of an engineering organization is to efficiently, in terms of time and resources, produce software according to the product planning and other organizational objectives and constraints (Fitzgerald and Stol 2014).

• Level II: Product planning organization analyzes various inputs and sets objectives for further product development. The planning organization’s primary objective is to prepare plans on leveraging software engineering to attain strategic objectives, e.g., market share, profitability, and outperform competition (Fitzgerald and Stol 2014; Humble and Kim 2018).

• Level III: Operations organization deploys and provides the software to the end-users. The primary objective of product operations is to deliver quality service to end-users (Fitzgerald and Stol 2014).

• Level IV: End-user organization uses and benefits from the software. The primary objective of this stakeholder is to maximize perceived value from the software (Fitzgerald and Stol 2014; Boehm 2003).

These stakeholders can be organized into different organizational structures. The organizational configuration is important because more involved organizations imply more boundaries, need to synchronize different paces, aiming for different objectives, unclear areas of responsibility, and other potential impediments to end-to-end continuous software delivery (Serrat 2017; Romano Jr et al. 2010).

The conceptual model shows an idealized state-of-the-art scenario. Analysis of the two industrial cases shows that adoption of the pipeline components varies, see Table 9. We elaborate details of the cases in Sections 4.1 – 4.2. Further sections elaborate state-of-the-art of each component.

Table 9 Mapping between the steps of CI/CD, state-of-the-art, and state-of-practice

4.4.1 Continuous Planning

Continuous planning, see Step #1, in Fig. 2, refers to a holistic activity involving multiple stakeholders to create lightweight, dynamic, and open-ended product plans. The planning is aimed to swiftly address and adjust to new market opportunities, changes in a business environment, and technologies (Fitzgerald and Stol 2014; Lehtola et al. 2009).

Continuous planning depends on access to immediate customer feedback and product telemetry to support decision making and organizational flexibility to make necessary adjustments rapidly, see Table 10. Without access to information, the planning activity falls short of delivering precise responses. Lack of organizational flexibility hinders the implementation of the plans as they may become outdated before the organization is able to react (Provost and Fawcett 2013; Li et al. 2010).

Table 10 Assumptions associated with continuous planning

The benefits of continuous planning concern faster response time to any inputs, thus helping the organization to be more flexible, see Table 11.

Table 11 Benefits associated with continuous planning

Investments in continuous planning concern the costs of more frequent analysis, planning, and coordination, see Table 12, and investments in data collection and analysis, see Table 26.

Table 12 Investments associated with continuous planning

4.4.2 Continuous Development

Continuous development, see Step #2 in Fig. 2, comprises activities to produce software. Development starts by receiving plans or tasks from the product planning organization, turning these plans into working software, and returning a ready-to-be-deployed software.

The continuity of development is achieved by minimizing any waiting in the process, for example, downtime until test results are in or shelving completed features until specified release date. Another important characteristic of continuity is to minimize dependencies between tasks and developers, enabling scalability through parallelization of development tasks (Humble and Kim 2018).

The literature identifies several activities that constitute continuous development.

Continuous Verification

see Step #2.1, in Fig. 2, refers to running various test suites frequently and in parallel with development. Practices such as using code linters (Tómasdóttir et al. 2017), following test-driven development, and running unit tests (Williams et al. 2003; Stolberg 2009) help to reduce the time between a defect are introduced and discovered. Source code analysis tools and pull-request review process help to ensure that the code conforms to the best practices and organizational standards (Shahin et al. 2017; Jiang et al. 2017)

Continuous integration

see Step #2.2, refers to frequent integration of software components and running of higher-level tests to ensure software as a whole work as intended and enabling continuous deployment, see Step # 3, (Pinto et al. 2018; Kim et al. 2008; Meyer 2014; Hilton et al. 2017; Hilton et al. 2016; Felidré et al. 2019).

Continuous Architecture

see Step #2.3, refers to a frequent assessment of software architecture to control software decay and adjusting the architecture to suit new and evolving scenarios (Del Rosso 2006; Riaz et al. 2009). The right software architecture is essential to support other CI/CD activities. For example, modularization of software reduces the need to coordinate between teams working on different parts of the software, simplifies verification, and enables quick deployment, Steps 3-4 in Fig. 2, of a part of the system (O’Connor et al. 2017; Humble and Kim 2018; Chen 2018).

Continuous Configuration Management

see Step #2.4, refers to a set of practices to ensure that all assets and routines for building and running software are versioned and scripted. The practices include source code versioning, isolating software instances and their dependencies in containers, build scripts, automated deployment to dedicated build/staging environment, and so on (Hasselbring and Steinacker 2017).

Continuous Non-functional Testing

see Step #2.5, refers to frequently verifying non-functional aspects of software (Yu et al. 2020).

State-of-the-art identifies numerous benefits of continuous development. Most benefits arise from implementing a robust and automated process for software verification. Full automation requires software to follow certain enforced standards (such as modularization, testability, unit test coverage). As a result, the overall software quality increases, and less manual effort is needed to make sure software works as intended. Thus, providing engineers with the flexibility to dedicate their time to more value-adding activities, see Table 14.

Table 13 Assumptions associated with continuous development

Table 14 Benefits associated with continuous development

Beneficiaries include developers who experience increased productivity and support for their tasks, the team experiencing improved culture and collaboration, product improving internal and external quality aspects, and the organization.

On the investments side, there are substantial investments to create automated test suites, integration environments, tooling, and test data. However, state-of-the-art offers limited perspective into estimating the costs of test automation and test data management, see Table 15.

Table 15 Investments associated with continuous development

Continuous development relies on test automation. Thus the ability to automate most, if not all, QA steps is a prerequisite, see Table 13.

4.4.3 Continuous Deployment

Continuous deployment, see Step #3 in Fig. 2 refers to a practice to continuously deploy the latest software to a staging environment and keep it ready for immediate delivery. The continuous deployment follows continuous development when software is integrated and passes all tests (Fitzgerald and Stol 2017).

In the literature, we found that terms delivery and deployment are sometimes mixed or used interchangeably, see, for example, Fitzgerald et al. (Fitzgerald and Stol 2014), and Neely et al. (Neely and Stolt 2013). We differentiate between the two. We use the term deployment to refer to internal readiness to deliver software to customers at any moment. With the term delivery, we refer to the ability to deliver the latest software to end-users at any time.

Continuous deployment relies on continuous integration and test automation at the earlier continuous development step, see Table 16.

Table 16 Assumptions associated with continuous deployment

The main benefits of continuous deployments stem from their frequent and incremental nature. Minor changes are easier to verify than large updates, reducing the risk of introducing severe issues. Automation reduces stress and overtime of preparing a release, thus increasing developers’ job satisfaction, see Table 17.

Table 17 Benefits associated with continuous deployment

The investments of continuous deployment concern the development of a robust acceptance test suite, and adjusting the organization. Continuous deployment requires downstream stakeholders (operations, customer support, marketing, etc.) to work with a versionless and continuously evolving product. This requires additional coordination effort, see Table 18.

Table 18 Investments associated with continuous deployment

4.4.4 Continuous Delivery

Continuous delivery, see Step #4, in Fig. 2, refers to making the latest features available to end-users immediately, i.e., with automated software upgrades. The critical difference between release-based and continuous delivery is that completed features are shelved until the scheduled release date in a release-based process. While shelved, a feature does not generate value (e.g., profit for the vendor and value for the customer), and it is uncertain to what extent the feature is relevant in the market. However, in continuous delivery, completed features forgo automated integration, verification, and acceptance steps are immediately delivered to the end-users. Thus, new features start generating value and feedback to steer further product development (Humble and Kim 2018; Chen 2015).

Continuous delivery assumes that the software vendor has access to the product for upgrades, and customers are willing to upgrade without notice, see Table 19. Renegotiating agreements with customers and establishing control over production environments are the key investments in adopting continuous delivery, see Table 21.

Table 19 Assumptions associated with continuous delivery

The primary benefits of continuous delivery stem from pushing the latest features to customers with no delay. Thus, increasing the value of the features and starting to collect customer feedback. In Table 20, we summarize the benefits from literature.

Table 20 Benefits associated with continuous delivery

Table 21 Investments associated with continuous delivery

Table 22 Assumptions associated with continuous use

4.4.5 Continuous Use

The shift from transactional, release-based software deliveries to continuous, versionless software highlight the importance of delivering software that satisfies customer needs over time, not just at the moment of purchase. Business models, such as software-as-a-service, monetize software, i.e., continuous use, not just its purchase. These developments create opportunities for software vendors to establish synergy with customers, improve understanding of their needs, and build trust. Mutual trust enables organizations to open up to each other (e.g., by sharing details on how the product is used and jointly developing new experimental features) (Fitzgerald and Stol 2017; Susarla et al. 2009).

The primary benefits from continuous use arise from closer, longitudinal relationships with customers and end-users. The closer relationship enables more opportunities for feedback collection, builds trust, and improves overall customer satisfaction, see Table 23.

Table 23 Benefits associated with continuous use

The investments of continuous use concern costs associated with maintaining longitudinal customer relationships and analyzing customer feedback. We differentiate between customer feedback arising from the use of the product, e.g., social media posts and app reviews, revealing experiences with the software, and product telemetry (discussed in the following subsection) capturing how the software is used, see Table 24

Table 24 Investments associated with continuous use

The benefits from continuous are relevant if the product offers features that are intended for continuous use. Software that is used rarely, e.g., a system that is used once a year to generate a yearly report, may not generate meaningful feedback or customer trust, see Table 22.

4.4.6 Continuous Monitoring

Continuous monitoring, see Step #5 in Fig. 2 is a practice to collect and analyze data on how the product is used and performs in a live environment, including metrics from the CI/CD pipeline. This data used in planning, monitoring helps to fine tune the product, CI/CD procedures, discover defects and quality issues early (Ehlers et al. 2011; van Hoorn et al. 2009)

There are no immediate benefits from collecting the data. Data collected at this step of the pipeline enables and supports continuous planning, and improvement activities, see Steps #1 and #7.6 in Fig. 2 (Olsson and Wnuk 2018; Johnson et al. 2005). Product usage data helps to prioritize test cases and to synthesize realistic test data in the continuous verification step, see Step #2 in Fig. 2, (Anderson et al. 2019).

Monitoring and data collection are associated with investments in setting up and maintaining relevant infrastructure, see Table 26. However, to implement continuous monitoring, customers should be ready to provide access to product usage data, see Table 25.

Table 25 Assumptions associated with continuous monitoring

Table 26 Investments associated with continuous monitoring

4.4.7 Cross-Cutting Concerns

Some pipeline components are rather cross-cutting than a discrete step in the software delivery pipeline, see components #7.* in Fig. 2. The benefits of these cross-cutting components include enable and support the rest of the pipeline, e.g. by generating data, setting performance indicators, and ensuring compliance to relevant regulations (Chen 2015; Shahin et al. 2017).

Primary investments in cross-cutting concerns tearing down existing organizational structures, business models, and ways of working to implement continuous software delivery pipeline throughout the organization, see Table 27.

Table 27 Investments associated with cross-cutting concerns

Continuous Compliance

see Step #7.1, in Fig. 2, refers to the practice to ensure compliance to relevant regulations continuously and throughout the pipeline, in contrast to compliance verification bottleneck at the end of development phase (Fitzgerald and Stol 2014; Moyon et al. 2018). Furthermore, agile and continuous principles are often at odds with regulatory practices. Practicing continuous software delivery in regulated environments requires a careful balance between speed and discipline (McHugh et al. 2013; Fitzgerald et al. 2013)

Continuous Security

see Step #7.2, in Fig. 2, refers to practicing security throughout the pipeline (Moyon et al. 2018; Dännart et al. 2019)

Continuous Budgeting

see Step #7.3, in Fig. 2, highlights the need for the whole organization to adopt continuous practices. Budgeting traditionally results in yearly, quarterly, etc., budgets tied to attaining specific objectives delegated to specific organizational units. However, such a plan-driven approach may hinder cooperation, speed, and flexibility associated with continuous practices (Frow et al. 2010; Lohan 2013).

Continuous Innovation

see Step #7.4, in Fig. 2, refers to a process throughout the pipeline to respond to evolving market conditions (Cole 2001; Olsson et al. 2012).

Continuous Experimentation

see Step #7.5, in Fig. 2, refers to a controlled, data-driven process of devising hypotheses about user preferences, setting experiments, and analyzing the results to drive product decisions (Fagerholm et al. 2017; Bosch 2012).

Continuous Improvement

see Step #7.6, in Fig. 2, arises from Lean principles and aims to minimize waste and perfecting the process. The improvements are achieved by process mapping, collecting metrics, observing performance indicators, and making minor adjustments throughout the pipeline (Cole 2001; Poppendieck et al. 2011).

5 Discussion

We have analyzed two cases to understand how they adopt CI/CD and their perspectives on the investments and benefits of CI/CD practices.

Our results show that in both cases, the adoption of CI/CD practices started by engineering teams wishing to improve internal efficiency. Teams had adopted automated test, build, and integration practices to aid their day-to-day work. There are immediate benefits of adopting such practices confirmed both by our cases, see Sections 4.1.4, 4.2.4, and literature, see Section 4.4.2. However, in the cases there are no systematically collected objective data reflecting the gains.

Both studied organizations report that implementing continuous deliveries to customer environments would be a significant challenge. Existing customer agreements are designed around planned releases and do not support continuous software deliveries. Deliveries to customer environments require adhering to customer-specific differences and dependencies. Furthermore, pushing customers to accept new terms of service create a risk of customers switching over to competitors’ offerings. The challenge is further exacerbated by the fact that both companies offer software to other organizations to provide services for end-users. The limited control over the software delivery pipeline hinders the speed of software delivery and limits exchange of data and feedback, especially in Case I, see Section 4.1.6.

Humble et al. (Humble and Kim 2018) argues that delivering minor frequent incremental updates alleviates the release pain. However, our interviewees responded that while they find the statement plausible, they lack concrete arguments and leverage to force their customers to accept continuous software deliveries. A specific challenge arises from customers’ developments on top of the vendor’s software. Any update introduces a risk of breaking the intricate dependencies. For this reason, accepting software updates involves extensive testing and custom development on the customer’s side. There is a potential to reduce the risk by synchronizing the development cycles between the customer and the vendor and strictly defining the interfaces. However, persuading many other downstream organizations to change and align their processes with the software vendor is unrealistic in the foreseeable future, see Sections 4.1.8 and 4.3.

Both organizations have chosen to adopt CI/CD due to ongoing high-level initiatives to improve software delivery. However, they have performed only a superficial analysis of what exact goals they wish to achieve and whether CI/CD is the most suited approach. More in-depth analysis is needed to understand how the implementation of CI/CD can support organizational objectives.

We continue the discussion by answering our research questions.

5.1 What are the Key Steps of the CI/CD Pipeline?

Literature suggests that an end-to-end continuous software delivery pipeline comprises planning, engineering, deployment, delivery, use, feedback collection, analysis, and other steps, see Fig 2 and Section 4.4. However, to our best knowledge, a complete implementation of an end-to-end CI/CD pipeline is yet to be demonstrated by empirical studies. State-of-the-art discusses parts of the pipeline and primarily focus on development, integration, and delivery steps.

In our studied cases, the scope of the CI/CD is limited to engineering, integration, deployment, and delivery steps. Practices such as continuous data collection, planning, and improvement are appealing, however currently unattainable, see Sections 4.1.8 and 4.3. Realizing the difficulties of achieving continuous delivery, organizations have not considered further steps.

It could be that end-to-end CI/CD as presented in our model, see Fig 2, and specific steps following continuous deployment are not feasible for products with a high level of customer customization, strict service level agreements, and a general push-back from existing customers. Furthermore, it could be that end-to-end CI/CD requires a special context granting the software vendor control over the pipeline and access to customer data.

Fitzgerald et al. (Fitzgerald and Stol 2017) argue that agile software development methodologies, such as scrum, are finding their way into regulated domains. Nevertheless, literature offers little support in retrofitting existing organizations with CI/CD capabilities. In particular, organizational interfaces such as business models, software delivery models, customer agreements are assumed to be flexible and adjustable without cost.

As shown by our two cases, organizations can benefit from automating and streamlining the internal development process without considering the complete end-to-end CI/CD. Thus, the focus on adopting CI/CD in complex domains could be to maximize the benefits from internal automation steps first, see Table 9.

Importantly, our empirical findings show that companies acknowledge the potential benefits end-to-end CI/CD pipeline as presented in Fig. 2, and Tables 4–7. However, when retrofitting existing offerings with CI/CD, organizations are cautious to make sweeping changes and to take risks that can disrupt their business. For instance, requiring customers to accept continuous deliveries and share data may backfire if customers are not prepared and ready to comply. Thus, the investments (costs) and associated risks nullify the potential benefits to adopting the practices, see Section 4.1.7.

5.2 What Costs and Benefits Steer the Adoption of the CI/CD in Practice?

In Ericsson’s case, the primary driver to adopt CI/CD is to streamline and speed up internal software delivery processes. The main challenge to solve is integrating a large number of software components into customized customer solutions efficiently, see Section 4.1.4.

The secondary objective is to improve their business models based on CI/CD capabilities. However, what exactly are these capabilities, the extent to which they are relevant to current customers, and how to retrofit current offerings to be compatible with the new business models remains unknown. As a consequence, the organization currently focuses on internal benefits, see Section 4.1.6.

In Telia, the only driver for adopting the CI/CD practices is to improve the internal processes and free up resources for more value-adding activities, see Section 4.2.4. While the organization realizes the potential benefits of continuous software deliveries, attaining them is not their current agenda.

During our workshops, the participants reflected that engineers generally welcome test and build automation initiatives, containerization, and other technical practices. The value of introducing such practices is immediate and substantial. However, the challenge is to convince management to invest in other CI/CD practices spanning multiple organizational units and organizational functions especially, if that requires adjustments in currently operational business models. For instance, pushing customer to accept more frequent deliveries and share data, see Tables 4–7.

Participants expressed that they realize the potential of developing new collaborative business models based on CI/CD capabilities. However, due to their organizational complexities and potential customer push-back, changing their business models on a scale is beyond their current planning. More analysis is required to understand how changes in business models could help to attain the organizational objectives and what are the constraints of such changes, see the discussion on Sections 4.1.6 and 4.3.

5.3 What is Needed to Support the CI/CD Cost-Benefit Analysis?

In both studied cases, we observed a difficulty to extend the continuous pipeline outside the development organization to customers and end-users. The software delivery process is an important feature and changing it changes the total value of the offering (Khurum et al. 2013). The two cases demonstrate that predictability and risk minimization in software deliveries takes precedence over delivery speed in domains where software is business-critical, see Sections 4.1–4.2.

The literature emphasizes that delivery speed and frequency is a solution to nearly all software delivery challenges, see for instance Humble et al. (Humble and Kim 2018). However, there is little discussion on weighing the benefits of schedule-based software releases versus faster access to new features from customers’ perspectives. For example, in customers’ eyes, the fact that software does not change could be an important feature if a software upgrade requires updating many intricate dependencies.

Avenue for further work: There is an opportunity to explore the mapping between software value aspects (Khurum et al. 2013) and benefits and trade-offs of continuous engineering. That is, knowing what value aspects are prioritized by customers and the software vendor enables a more fine-grained analysis of how continuous engineering contributes to these aspects.

We observe that organizations are often not aware of what specific goals they aim to attain by adopting CI/CD, becoming better in delivering software and what are the constraints of any solutions they wish to implement. Attaining one goal, e.g. increasing speed by implementing automation and eliminating manual steps, could imply more standardization. However, that can be counterproductive if customers expect individual treatment and bespoke solutions. This issue is evident in Case I, see Section 4.1.6, where the organization contemplates the benefits of new collaborative business models, however there is not enough details to gauge the relevance of such models in their domain.

Avenue for further work: There is an opportunity to devise a methodology to identify and break down organizational goals towards software delivery and estimate the degree of freedom to implement any changes. The adoption of any new practices or tools should be guided by specific and measurable goals. Importantly, the software delivery pipeline transcends individual organizational departments and extends into a customer organization. Therefore, the goals, KPIs, and objectives should aligned and shared across the organization and accepted by the customers.

More understanding of the organizational benefits is needed to support management decisions concerning adopting CI/CD and driving the necessary organizational changes. Both engineering and organizational literature, see Section 4.4 and, e.g. Hanelt et al. (Hanelt et al. 2021), discusses organizational benefits such as cooperative business models and telemetry to tailor the offering to specific customer needs. However, it is unknown how to gauge relevance, quantify, and compare these benefits to the required investments.

Avenue for further work: The relationship between digital transformation and continuous engineering in software organizations need to be explored. Digital transformation explores the organizational benefits of adopting technology to streamline their processes. Continuous engineering focus on the benefits from streamlining software engineering. Exploring how one could enable and support another could help establishing a joint management-engineering view on the associated changes, benefits and required investments.

We further notice that the boundaries of control constrain the adoption of CI/CD. In the studied cases, the software vendors only partially control the delivery pipeline. By design, the final installation and release to end-users is out of control of the software vendor, see Section 4.1.3. The literature overlooks this and assumes that the software vendor has complete control over the pipeline. Expanding control boundaries and assuming more responsibilities requires substantial changes in the offering and the underlying business model. It also implies reducing the level of control other stakeholders has over the software. More understanding is needed to gauge the implications of extending boundaries of control.

Avenue for further work: More research is required how to integrate different pipelines of software delivery with potentially different aims, release cadences, stakeholders, and controlled by different organizations. Moreover, large organizations, such as Ericsson, may have multiple internal pipelines that could branch and merge in different ways and are controlled by different internal stakeholders. It remains to be explored how to streamline and synchronize software delivery in such scenarios.

6 Conclusions

This paper has examined the adoption of continuous software engineering practices in two industrial cases and performed a literature study to understand the costs and benefits perspective on the CI/CD.

We found that literature overstates the benefits of CI/CD without recognizing specific domain complexities and the challenges of retrofitting already existing offerings with CI/CD capabilities. Importantly, literature overlooks the customer perspective on accepting continuous software deliveries and provides little guidance on upgrading existing customers to continuous mode.

The two case studies reveal that the adoption of CI/CD in organizations starts from the bottom-up engineering teams attempting to automate and streamline their work. However, expanding the continuous software delivery pipeline to adjacent organizational units, especially to customer organizations, is challenging. A significant challenge is to gauge the relevance of CI/CD benefits for a specific organizational domain and context.

We identify a need to explore customer further and organizational perspectives and understand the contextual requirements for adopting continuous software engineering practices. Furthermore, we identify a gap in state-of-the-art concerning retrofitting an existing product with a CI/CD pipeline.