To understand drivers behind the adoption of CI/CD practices, we study state-of-practice in two companies and map our findings with state-of-the-art. Using this mapping, we highlight parts that overlap, are missing in state-of-the-art, but also items overlooked by practitioners in state-of-the-practice. We differentiate between 5 types of mapping points, see Table 2.
We structure our results in two parts. In the first part, we present two industrial cases highlighting their contexts, wants, and needs in relation to CI/CD. In the second part, we present state-of-the-art perspecitve, connect it to the state of practice to identify gaps, discrepancies, and relevant results to support the adoption of CI/CD in practice.
Case I: Ericsson
Ericsson AB is a large telecommunications hardware and software provider based in Sweden. The company offers an extensive portfolio of software-intensive products. They combine their offerings with 3rd party products to create customer-specific solutions. Customer solutions are highly customized and delivered to mobile network service providers globally.
In this study, we focus on a product aimed to support mobile telecommunications operators’ business functions (Product A). It is a large and complex constellation of different components. Individual software components have a varying degree of continuous integration and deployment capabilities. However, the challenge lies in the solution-level integration and testing. Differences between components, individual component delivery methods, customer-specific customizations, dependencies on third-party components, and domain regulations make final software integration complex, slow, and manual effort-intensive.
Our primary contact in Ericsson was a service delivery organization. The organization is responsible for orchestrating Ericsson’s different activities to attain efficient software deliveries to customers and align the delivery process with their needs. Recently, they have started an initiative to create a strategy for using CI/CD to streamline the software delivery process, focusing on implementing a universal pipeline for delivering the Product A.
We organized the data collection activities in Ericsson between September 2020 and March 2021, see Table 3. In addition to meetings and workshops, Ericsson provided access to a large number of slide decks containing information about their ongoing work on streamlining software delivery. We used these resources to prefill our inventory spreadsheet and steer discussions in the meetings.
Due to the size of the product and large number of stakeholders involved, the inventory spreadsheet was filled in incrementally. Stakeholders were able to provide information only in their respective areas of responsibility. In one of the final workshops we gathered a larger group of stakeholders to discuss the overall picture. We also jointly developed a mind map capturing the organizational goals, links to specific engineering challenges to be solved, pertinent continuous practices, and obstacles on implementing the practices.
We summarizing the insights from Ericsson illustrating their case in the following subsections. We discuss the mapping between our empirical findings and state-of-the-art in Section 4.4.
Steps of the Software Delivery
The software delivery in Ericsson is a multi-stage process. It can be summarized as follows:
Individual components are developed using a varying degree of continuous development practices. The latest components are placed in a repository for further integration.
Service delivery organization combines different components into customer solutions. Frequently, customer solutions contain bespoke customizations.
Customer solutions are deployed in a staging environment where both software vendors and customers verify the software.
Once accepted, a solution is taken to customers’ premises for further validation, configuration, and integration with customers’ systems. It may take several months and up to a year until an accepted solution goes live.
There is a predefined update schedule determining when customers should be prepared for upgrading their solution. The interval between upgrades spans multiple years.
More often than not, transitions between steps of the software delivery are a pull rather than push operation. The reasons include ensuring control and stability of staging and operational environments and service level agreements between the vendor and customers. For similar reasons, verification of releases, especially regarding compliance and non-functional qualities, is performed manually and as a separate step before deployment.
Aim I - Rapid Delivery of New Technologies
Ericsson aims to remain on top of the market with the rapid delivery of innovative offerings. Emerging technologies such as 5G will require software vendors to react more rapidly to new market needs and provide innovative solutions on short notice. The current release pace is not sufficient to address this objective. Specifically, the preparation (by vendor) and adoption (by customer) of releases are overly time-consuming, complex, and expensive. As one interviewee reflected:
“With 5G we must be much faster in rolling out new services. The technology is going to be much more dynamic. We can’t wait 6 months to deliver something and then 6 more months to get feedback.”
To address this objective, the study participants identify two areas of improvement. Firstly, rapid software delivery depends on streamlined development supported by automation—secondly, efficient solution level integration and deployment to staging environments.
Continuous development is practiced to a significant extent. Test, build, and low-level integration is largely automated already. However, due to compliance requirements and legacy processes, some verification steps are performed manually and create a bottleneck at the end of the development phase.
Solution level integration, development, and staging are implemented to deliver highly customized solutions and to jointly with a customer to verify that a release is stable and reliable for adoption. Much time is spent ensuring the reliability of the release and waiting for the proper launch window to upgrade customers’ systems.
The customer systems are often piecemeal solutions consisting of various components from various vendors and in-house developments. Thus, upgrading part of a system may require upgrades and additional developments in 3rd party components. An important part of Ericsson’s offering is to provide solutions that fit within existing customer systems. There is an ongoing initiative to explore to what extent the solution automation can be streamlined and automated. As an interviewee described customer solutions:
“Customers usually develop their systems over time and look for what is more economical for them. A typical solution contains some of our components, components from other vendors, and some of their own software. Ericsson puts in a lot of effort to make sure our components are compatible with whatever customers have built.”
Ericsson needs to implement and streamline their continuous development, deployment, and, potentially delivery, practices to fulfill this aim. However, state-of-the-art offers little support for practicing CI/CD in complex regulated environments; see rows 1–3 in Table 4.
Aim II - Universal Pipeline to Deliver Software
There is an aim to simplify and unify the software delivery process for different offerings and customers. The current software delivery process is tailored to the specifics of individual offerings and the needs of specific customers. Thus, there is little standardization and, consequently, much room for optimization.
Interviewees identify that the product architecture, for example, component scopes and interfaces, are not optimal for continuous solution level integration. The architectural shortcomings increase solution complexity, thus making efficient integration and delivery difficult. As described by a workshop participant:
“We have a “red team” of our top engineers. Initially it was planned only for incident response. However, they are involved all the time whenever we are preparing a release.”
To remedy this, Ericsson needs to evolve their architectures with CI/CD in mind. Literature suggests that cloud-native components with few interdependencies perform best. However, to what extent investments in evolving existing products to fit well with CI/CD can be justified is unknown.
Aim III - New Business Models
More frequent software deliveries and access to feedback could enable closer cooperation between Ericsson and customers. Thus, enabling new collaborative business models.
Our study participants reflect that this is a very vague aim that needs more details to judge whether it is relevant in the telecom domain. Further discussion revealed that developing new business models must support the existing offerings until all customers are upgraded to the new models. Some customers may never accept continuous deliveries. This creates a potential situation when the organization offers the same product with continuous deliveries and plan-driven software releases. Quoting one participant:
“Sure, it sounds exciting! However, telecom domain has been very slow in adopting innovations in business models. Largely, because we do what operators want and some operators do not really want any innovation.”
Support for parallel business models and software delivery methods adds new requirements and complexity throughout the organization. For example, software must be developed with support for both the old and the new delivery method. Both delivery methods need to be synced to ensure consistency in terms of product features and quality. Customer support must be prepared to assist in both scenarios.
State-of-the-art assumes that the product is always delivered continuously (or release-based) and offers little support for transitioning from one model to another, see row 6, in Table 4.
Aim VI - Data-Driven Decision Making
Ericsson aims to benefit from data-driven techniques in improving both internal processes and customer offerings. To attain this objective, Ericsson needs to implement metrics to measure the software delivery process and establish means of getting access to product telemetry.
The study participants reveal that there have been attempts to gain access to product telemetry and usage data. However, customers are unwilling to share any data to protect their business secrets and the privacy of end-users. As one participant reflected:
“We have had endless discussions to get access to some telemetry. However, our customers see that as a risk rather than an opportunity.”
Ericsson has only partial control over the software delivery pipeline. The software vendor controls the pipeline internally until the latest release is ready for delivery. However, the vendor has limited control of whether customers upgrade to the latest version and no control or access to software once it is operational. One interviewee shared an old but relevant anecdote:
“Once in the ninethies we delivered a new solution with both hardware and software to an operator. It cost them a lot. Our sales teams tried to follow up and learn how satisfied the customer is. However, no useful information came back. Finally, six months later we sent a representative to investigate. It turned out that the operator had not even unpacked the boxes yet.”
Recently with the adoption of cloud-based solutions, the situation has improved. However, the challenge of gathering data from customers is still relevant.
What is Needed to Support Retrofitting Product Product A with CI/CD Pipeline?
Analysis of Case I shows that retrofitting a product with a CI/CD pipeline is a substantial undertaking and introduces many business risks in addition to technical challenges. For example, from customers’ perspective switching to a new product release model and renegotiating service agreement may trigger consideration of alternative offerings by competitors.
Comparing lessons from state-of-the-art and Case I, see Table 4, we identify the following areas for support.
Component Granularity and Flexibility
Current granularity of components is not well suited to support different configurations of customer solutions. Thus, there is additional bespoke development at each release to tailor the solution to customer needs, see rows 1-4 in Table 4.
Component boundaries and limits of flexibility need to be revised to streamline the release process and keep bespoke development at minimum.
Support for Different Parallel Delivery Models
Not all customers could be ready for changes in the product and its delivery method. Customers have built their own and tightly coupled solutions on top of Ericsson’s offerings. Thus, any changes in architecture, components, interfaces, etc., will have a substantial impact, see rows 1–2,4–6 in Table 4.
Currently, customers rely on a predefined release pace with ample time to prepare other systems and their business processes for integration with the new release. Thus, different customers may require different release paces and extent of backward compatibility. Consequently, Ericsson must be prepared to roll out any changes in components and release pace softly while maintaining full support for customers who do not wish to change right away.
Ericsson needs to balance investments, risks, and potential benefits to decide whether it is worth the investment to change the product architecture and upgrade current customers to the improved software delivery model. We summarize the decision in Table 5.
Our interviewees state that the organization has little appetite for the risk of losing existing customers. Thus, the focus is set to adopt CI/CD with minimal disruption to customers. However, the objectives to change the software delivery model without affecting customers are contradictory. More understanding of the customers’ perspective and willingness to switch to continuous software deliveries is needed.
Case 2: Telia Company
Telia Company AB is a telecommunications services provider in Sweden. The company provides telecommunications services to private and business customers. To serve its customers, the company integrates in-house built software with off-the-shelf products.
The current release pace is slow due to domain complexities and dependencies on other systems with long release cycles. However, there is an ongoing initiative to improve internal efficiency and speed up software releases. The initiative is not specific to CI/CD. However, continuous software delivery is considered to be one of the candidate solutions.
The company had selected one of their products (from now on, Product B) as a pilot case for adopting continuous engineering practices. The product is a software layer between other systems supporting an exchange of text messages. The product is stable, mature, and most developments concern maintenance updates and occasional bug fixes. The product is not exposed to customers directly. Instead, it enables features for other products with user interfaces.
We conducted several workshops with the product team to understand their objectives, motivation, constraints, and challenges in adopting CI/CD practices. The product manager, two senior developers, delivery manager, and operations specialist where the core participants. The data collection took place between September, 2020 and April 2021, see Table 6.
Steps of Software Delivery
Product B is relatively small and developed by a loose team of few developers. Other stakeholders include an operations team responsible for deploying and ensuring the smooth operation of the product. The steps of delivering Product B are:
Developers receive issue tickets from the operations team and implement necessary changes. The flow of tickets is slow, and developers prepare two releases a year.
Finished software is complemented with extensive instructions for deployment and operation and forgo several rounds of testing.
Once a release is ready, operations teams take over and deliver the release to customers with support from the development team.
Aim I - Simplify the Development and Build Process
Due to strong coupling with other systems, Product A requires a complex environment for development and testing. Setting up the local environment and ensuring the correct configuration for development and testing is a complicated task. The team aims to benefit from containerization, build scripts, and similar practices to alleviate software dependency management issues. As one participant summarized the challenge:
“It used to take about two weeks to set up a development environment from scratch. The product has many intricate dependencies that are documented nowhere. Most of the time is spend on troubleshooting compatibility issues. With Docker containers we do not have this issue anymore.”
The slow-release pace of two releases per year has created an issue of knowledge preservation. Over long periods of inactivity on Product B, developers are assigned to other tasks or may have left the company. Every small change requires context switching and re-learning the specifics of the product. As a consequence, developers spend substantial time reading documentation and setting up individual development environments.
Aim II - Reduce the Complexity of Software Upgrades
Currently, every release of Product B is complemented with extensive documentation. Preparing this documentation is a substantial undertaking. The exact audience and value of the documentation are unclear to the development team. However, some documentation (e.g., installation instructions, test cases) is required as software artifacts are passed from development to operations. As described by one interviewee:
“Deployment of the product is not straightforward and operations often need our support to get things going even with all the documentation we provide. I think with automated scripts and containers we can make deliveries easier and leave less room for issues.”
The software vendor aims to streamline software delivery process by tearing down organizational silos and closer collaboration between development and operations. Such a move should reduce the need for extensive documentation and streamline the development-operations process as there would be one team responsible for the whole process.
Current customer agreements determine the software delivery process and do not support frequent releases. Furthermore, customers are not eager to receive frequent software deliveries. They mostly contain maintenance updates with little perceived value and increase the risk of system outages and other issues. The software vendor aims to simplify the release adoption process from the customers’ viewpoint. One participant reflected on this:
“The upgrades does not provide much new features, thus there is little incentive to upgrade right away. We have to send a notice weeks in advance to let customers prepare for an upgrade. We know very little what it takes for them to upgrade.”
Aim III - Improve Software Quality
Currently, Product B forgo several rounds of testing in different test environments before a release. Although some tests are automated, there is considerable potential to benefit from automated testing and verification practices.
Some tests are difficult to automate because the tests are complex, expensive to run, and prone to distrust in automated testing practices. These tests remain as a bottleneck at the end of the otherwise automated integration and verification pipeline.
Due to the high turnaround of developers, the product had eroded and accumulated some technical debt. To support the refactoring effort, the team wishes to benefit from automated and continuous verification practices.
The software vendor aims to benefit from continuous integration and deployment practices to remove bottlenecks of manual testing, simplify the release process, and improve the overall quality of the product. As stated by one interviewee:
“We have started automating some tests and it shows results. However, we still have some pretty heavy manual tests at the end. It would be good to automate them as well. However, we feel that sometimes there is distrust in automated tests and stakeholders are more confident with manual testing.”
What is Needed to Support Retrofitting Product B with a CI/CD Pipeline?
Currently, there are no inherent obstacles to retrofitting Product B with a continuous integration pipeline. However, the team reflects that by adding new practices and ways of working, they need to revise and remove any obsolete activities. For example, multiple testing rounds on different environments are no longer relevant if the product is containerized. Similarly, deployment scripts can replace installation instructions.
Identification of obsolete steps in development, integration, and deployment is essential to avoid unnecessary investments in automation. However, changes in the software delivery process need to be communicated and explained to all stakeholders.
Our interviewees admitted that they mainly consider the engineering perspective. The interests of other stakeholders, namely customers and adjacent products, have not been considered. Further work is needed to understand software deliveries from customers perspective and to what extent continuous software deliveries reduces the effort to adopt a new software release.
Comparing reflections from participants with state-of-the-art, see Table 7, we observe that practice agrees with literature. Specifically, continuous practices reduces the complexity of software engineering and removes tedious setup and configuration steps, and reduces the need for detailed documentation.
Continuing adoption of CI/CD, needs to explore the interests of other stakeholders. For example, exploring the potential of aligning software delivery pipelines across the whole organization. Thus, creating a coherent software delivery pipeline that can be further extended to customers organizations. We summarize the investment-benefit conundrum in Table 8.
The CI/CD initiative in the Product B was intended to pilot continuous software delivery within the organization to gauge its further potential. Thus, attaining more than “just enough” continuous capabilities could be a worthy investment to gain a complete picture of the potential benefits, investments and pitfalls of CI/CD in their specific context.
Connection to State-of-the-Art
A substantial amount of research addresses individual building blocks of a CI/CD pipeline, e.g., test automation (Wiklund et al. 2017; Raulamo-Jurvanen et al. 2017; Kasurinen et al. 2010). However, a big picture perspective is needed to identify all relevant components of a pipeline (RQ1) and understand how individual components of a pipeline fit together to analyze their benefits (RQ2) (Frank 2000).
Inspired by Fitzgerald et al. (Fitzgerald and Stol 2017), we develop a conceptual model visualizing state-of-the-art CI/CD pipeline, see Fig. 2. Importantly, the model is based on state-of- the-art in the area. We use it as an overview and tool in our industry workshops to aid discussions of costs, benefits, relevance, and dependencies between different parts of the pipeline.
In the figure, we denote the main pipeline steps with rectangular blocks. With dashed boxes, we illustrate the key benefits and investments associated with each step. The flow through the pipeline is denoted with arrows. Components of the pipeline are organized by the four main levels of stakeholders in a CI/CD pipeline:
Level I: Engineering organization produces software. The primary objective of an engineering organization is to efficiently, in terms of time and resources, produce software according to the product planning and other organizational objectives and constraints (Fitzgerald and Stol 2014).
Level II: Product planning organization analyzes various inputs and sets objectives for further product development. The planning organization’s primary objective is to prepare plans on leveraging software engineering to attain strategic objectives, e.g., market share, profitability, and outperform competition (Fitzgerald and Stol 2014; Humble and Kim 2018).
Level III: Operations organization deploys and provides the software to the end-users. The primary objective of product operations is to deliver quality service to end-users (Fitzgerald and Stol 2014).
Level IV: End-user organization uses and benefits from the software. The primary objective of this stakeholder is to maximize perceived value from the software (Fitzgerald and Stol 2014; Boehm 2003).
These stakeholders can be organized into different organizational structures. The organizational configuration is important because more involved organizations imply more boundaries, need to synchronize different paces, aiming for different objectives, unclear areas of responsibility, and other potential impediments to end-to-end continuous software delivery (Serrat 2017; Romano Jr et al. 2010).
The conceptual model shows an idealized state-of-the-art scenario. Analysis of the two industrial cases shows that adoption of the pipeline components varies, see Table 9. We elaborate details of the cases in Sections 4.1 – 4.2. Further sections elaborate state-of-the-art of each component.
Continuous planning, see Step #1, in Fig. 2, refers to a holistic activity involving multiple stakeholders to create lightweight, dynamic, and open-ended product plans. The planning is aimed to swiftly address and adjust to new market opportunities, changes in a business environment, and technologies (Fitzgerald and Stol 2014; Lehtola et al. 2009).
Continuous planning depends on access to immediate customer feedback and product telemetry to support decision making and organizational flexibility to make necessary adjustments rapidly, see Table 10. Without access to information, the planning activity falls short of delivering precise responses. Lack of organizational flexibility hinders the implementation of the plans as they may become outdated before the organization is able to react (Provost and Fawcett 2013; Li et al. 2010).
The benefits of continuous planning concern faster response time to any inputs, thus helping the organization to be more flexible, see Table 11.
Investments in continuous planning concern the costs of more frequent analysis, planning, and coordination, see Table 12, and investments in data collection and analysis, see Table 26.
Continuous development, see Step #2 in Fig. 2, comprises activities to produce software. Development starts by receiving plans or tasks from the product planning organization, turning these plans into working software, and returning a ready-to-be-deployed software.
The continuity of development is achieved by minimizing any waiting in the process, for example, downtime until test results are in or shelving completed features until specified release date. Another important characteristic of continuity is to minimize dependencies between tasks and developers, enabling scalability through parallelization of development tasks (Humble and Kim 2018).
The literature identifies several activities that constitute continuous development.
see Step #2.1, in Fig. 2, refers to running various test suites frequently and in parallel with development. Practices such as using code linters (Tómasdóttir et al. 2017), following test-driven development, and running unit tests (Williams et al. 2003; Stolberg 2009) help to reduce the time between a defect are introduced and discovered. Source code analysis tools and pull-request review process help to ensure that the code conforms to the best practices and organizational standards (Shahin et al. 2017; Jiang et al. 2017)
see Step #2.2, refers to frequent integration of software components and running of higher-level tests to ensure software as a whole work as intended and enabling continuous deployment, see Step # 3, (Pinto et al. 2018; Kim et al. 2008; Meyer 2014; Hilton et al. 2017; Hilton et al. 2016; Felidré et al. 2019).
see Step #2.3, refers to a frequent assessment of software architecture to control software decay and adjusting the architecture to suit new and evolving scenarios (Del Rosso 2006; Riaz et al. 2009). The right software architecture is essential to support other CI/CD activities. For example, modularization of software reduces the need to coordinate between teams working on different parts of the software, simplifies verification, and enables quick deployment, Steps 3-4 in Fig. 2, of a part of the system (O’Connor et al. 2017; Humble and Kim 2018; Chen 2018).
Continuous Configuration Management
see Step #2.4, refers to a set of practices to ensure that all assets and routines for building and running software are versioned and scripted. The practices include source code versioning, isolating software instances and their dependencies in containers, build scripts, automated deployment to dedicated build/staging environment, and so on (Hasselbring and Steinacker 2017).
Continuous Non-functional Testing
see Step #2.5, refers to frequently verifying non-functional aspects of software (Yu et al. 2020).
State-of-the-art identifies numerous benefits of continuous development. Most benefits arise from implementing a robust and automated process for software verification. Full automation requires software to follow certain enforced standards (such as modularization, testability, unit test coverage). As a result, the overall software quality increases, and less manual effort is needed to make sure software works as intended. Thus, providing engineers with the flexibility to dedicate their time to more value-adding activities, see Table 14.
Beneficiaries include developers who experience increased productivity and support for their tasks, the team experiencing improved culture and collaboration, product improving internal and external quality aspects, and the organization.
On the investments side, there are substantial investments to create automated test suites, integration environments, tooling, and test data. However, state-of-the-art offers limited perspective into estimating the costs of test automation and test data management, see Table 15.
Continuous development relies on test automation. Thus the ability to automate most, if not all, QA steps is a prerequisite, see Table 13.
Continuous deployment, see Step #3 in Fig. 2 refers to a practice to continuously deploy the latest software to a staging environment and keep it ready for immediate delivery. The continuous deployment follows continuous development when software is integrated and passes all tests (Fitzgerald and Stol 2017).
In the literature, we found that terms delivery and deployment are sometimes mixed or used interchangeably, see, for example, Fitzgerald et al. (Fitzgerald and Stol 2014), and Neely et al. (Neely and Stolt 2013). We differentiate between the two. We use the term deployment to refer to internal readiness to deliver software to customers at any moment. With the term delivery, we refer to the ability to deliver the latest software to end-users at any time.
Continuous deployment relies on continuous integration and test automation at the earlier continuous development step, see Table 16.
The main benefits of continuous deployments stem from their frequent and incremental nature. Minor changes are easier to verify than large updates, reducing the risk of introducing severe issues. Automation reduces stress and overtime of preparing a release, thus increasing developers’ job satisfaction, see Table 17.
The investments of continuous deployment concern the development of a robust acceptance test suite, and adjusting the organization. Continuous deployment requires downstream stakeholders (operations, customer support, marketing, etc.) to work with a versionless and continuously evolving product. This requires additional coordination effort, see Table 18.
Continuous delivery, see Step #4, in Fig. 2, refers to making the latest features available to end-users immediately, i.e., with automated software upgrades. The critical difference between release-based and continuous delivery is that completed features are shelved until the scheduled release date in a release-based process. While shelved, a feature does not generate value (e.g., profit for the vendor and value for the customer), and it is uncertain to what extent the feature is relevant in the market. However, in continuous delivery, completed features forgo automated integration, verification, and acceptance steps are immediately delivered to the end-users. Thus, new features start generating value and feedback to steer further product development (Humble and Kim 2018; Chen 2015).
Continuous delivery assumes that the software vendor has access to the product for upgrades, and customers are willing to upgrade without notice, see Table 19. Renegotiating agreements with customers and establishing control over production environments are the key investments in adopting continuous delivery, see Table 21.
The primary benefits of continuous delivery stem from pushing the latest features to customers with no delay. Thus, increasing the value of the features and starting to collect customer feedback. In Table 20, we summarize the benefits from literature.
The shift from transactional, release-based software deliveries to continuous, versionless software highlight the importance of delivering software that satisfies customer needs over time, not just at the moment of purchase. Business models, such as software-as-a-service, monetize software, i.e., continuous use, not just its purchase. These developments create opportunities for software vendors to establish synergy with customers, improve understanding of their needs, and build trust. Mutual trust enables organizations to open up to each other (e.g., by sharing details on how the product is used and jointly developing new experimental features) (Fitzgerald and Stol 2017; Susarla et al. 2009).
The primary benefits from continuous use arise from closer, longitudinal relationships with customers and end-users. The closer relationship enables more opportunities for feedback collection, builds trust, and improves overall customer satisfaction, see Table 23.
The investments of continuous use concern costs associated with maintaining longitudinal customer relationships and analyzing customer feedback. We differentiate between customer feedback arising from the use of the product, e.g., social media posts and app reviews, revealing experiences with the software, and product telemetry (discussed in the following subsection) capturing how the software is used, see Table 24
The benefits from continuous are relevant if the product offers features that are intended for continuous use. Software that is used rarely, e.g., a system that is used once a year to generate a yearly report, may not generate meaningful feedback or customer trust, see Table 22.
Continuous monitoring, see Step #5 in Fig. 2 is a practice to collect and analyze data on how the product is used and performs in a live environment, including metrics from the CI/CD pipeline. This data used in planning, monitoring helps to fine tune the product, CI/CD procedures, discover defects and quality issues early (Ehlers et al. 2011; van Hoorn et al. 2009)
There are no immediate benefits from collecting the data. Data collected at this step of the pipeline enables and supports continuous planning, and improvement activities, see Steps #1 and #7.6 in Fig. 2 (Olsson and Wnuk 2018; Johnson et al. 2005). Product usage data helps to prioritize test cases and to synthesize realistic test data in the continuous verification step, see Step #2 in Fig. 2, (Anderson et al. 2019).
Monitoring and data collection are associated with investments in setting up and maintaining relevant infrastructure, see Table 26. However, to implement continuous monitoring, customers should be ready to provide access to product usage data, see Table 25.
Some pipeline components are rather cross-cutting than a discrete step in the software delivery pipeline, see components #7.* in Fig. 2. The benefits of these cross-cutting components include enable and support the rest of the pipeline, e.g. by generating data, setting performance indicators, and ensuring compliance to relevant regulations (Chen 2015; Shahin et al. 2017).
Primary investments in cross-cutting concerns tearing down existing organizational structures, business models, and ways of working to implement continuous software delivery pipeline throughout the organization, see Table 27.
see Step #7.1, in Fig. 2, refers to the practice to ensure compliance to relevant regulations continuously and throughout the pipeline, in contrast to compliance verification bottleneck at the end of development phase (Fitzgerald and Stol 2014; Moyon et al. 2018). Furthermore, agile and continuous principles are often at odds with regulatory practices. Practicing continuous software delivery in regulated environments requires a careful balance between speed and discipline (McHugh et al. 2013; Fitzgerald et al. 2013)
see Step #7.2, in Fig. 2, refers to practicing security throughout the pipeline (Moyon et al. 2018; Dännart et al. 2019)
see Step #7.3, in Fig. 2, highlights the need for the whole organization to adopt continuous practices. Budgeting traditionally results in yearly, quarterly, etc., budgets tied to attaining specific objectives delegated to specific organizational units. However, such a plan-driven approach may hinder cooperation, speed, and flexibility associated with continuous practices (Frow et al. 2010; Lohan 2013).
see Step #7.4, in Fig. 2, refers to a process throughout the pipeline to respond to evolving market conditions (Cole 2001; Olsson et al. 2012).
see Step #7.5, in Fig. 2, refers to a controlled, data-driven process of devising hypotheses about user preferences, setting experiments, and analyzing the results to drive product decisions (Fagerholm et al. 2017; Bosch 2012).
see Step #7.6, in Fig. 2, arises from Lean principles and aims to minimize waste and perfecting the process. The improvements are achieved by process mapping, collecting metrics, observing performance indicators, and making minor adjustments throughout the pipeline (Cole 2001; Poppendieck et al. 2011).