Keywords

1 Introduction

DevSecOps is an emerging approach to software development denoting integrated security controls and practices, and security teams, throughout the tasks of the development and system operations (DevOps) cycle [13, 14]. While the agile combination of development and operations as such was introduced more than a decade ago [6, 9], the integration challenge of security issues into agile development has continued in practice [15] and research [1, 14] alike. Reviews by Rajapakse et al. [14] and Akbar et al. [1] have outlined a great many challenges, 21 and 18, respectively, for DevSecOps adoption and management. While the industrial domain requires well-synchronized DevOps of software together with operational technologies (OT), the challenge of implementing secure coding standards, testing for security in DevOps and the sheer knowledge of role of security in connection to system and software development remain as the prioritized problem areas in the software industry [1].

In industries that involve cyber-physical systems, such as automation and control systems, information technology (IT) and OT need to be converged [4]. Such systems rely on the use and adoption of standards. The IEC 62443–4-1 standard focuses on cybersecurity during the development lifecycle especially in automation and control systems [7]. Although the standards in general form a basis to secure software development, their adoption, implementation, and operationalizing in practice is a time-consuming and laborious process. One of the core challenges of adopting DevSecOps for OT and related software is the very adoption of the often-complex security standards, such as IEC 62443–4-1, so that the professionals would also be able to operationalize the standard requirements in the development process synchronized with operations. Several issues related to this challenge are highlighted in [1, 10, 12, 14] but the empirical research on adoption and implementation of standards (e.g., IEC 62443–4-1), with actual software processes and tools, is still in its infancy [1].

Among the earliest research efforts on adoption of IEC 62443–4-1 in agile development of industrial systems, Moyon et al. [11, 12] suggest process models to be used collaboratively by security and development professionals to reach a common understanding on how to operationalize standard compliant DevSecOps. They [12] suggest that process/task-oriented understanding of the standard, indeed, becomes easier after modelling the resulting practices in the process form (with a business process modelling notation). While Moyon et al. [11, 12] provide, to our knowledge, the first demonstrations of the potential usefulness of their suggested approach, they do not report how and whether their process-based view has been operationalized in practice. Room for additional research on the security standard adoption challenge in connection to agile software development thus exists. Keeping this in mind, our research provides an early report on longitudinal experiences of actual adoption and consulting process for operationalizing IEC 62443–4-1 in the DevSecOps context of industrial automation systems.

Our research set out with a research question: How to operationalize the requirements of IEC 62443–4-1 security standards in agile DevOps of industrial automation systems? Our action design research (ADR) [16] effort covers four years of consultation and collaborative development for support practices and tools for standard adoption. Insta (https://www.insta.fi/en/en/) is a security consulting company working both in-depth and longitudinally with several customer organizations and cases simultaneously. In this research, Insta had an interest in developing practices and tool support for standard compliant DevSecOps. A main contribution to such formalized experience comes from Valmet Automation Systems (VAS), which is an early certified adopter of the IEC 62443–4-1 standard with its certified ISASecure® [8] SDLA (Security Development Lifecycle Assurance) process. VAS is a business line within Valmet corporation (https://www.valmet.com/automation/), which has co-operated with Insta over several years. This process included researchers from both Valmet and Insta, as well as a researcher from academia. The contributions of the paper can be summarized as follows:

  1. 1.

    Proposing three design principles for adopting and implementing IEC 62443–4-1 standard in practice.

  2. 2.

    Proposing two design objectives regarding portable process information model representation and process information model management tool.

  3. 3.

    Presenting a real-world case by Valmet Automation Systems where the DevSecOps process has been adopted and operationalized.

2 Methodology

The ADR [16] method focuses on co-operation between researchers and practitioners to create new knowledge. The ADR approach denotes that relevant research on IT artefacts benefits greatly from collaboration with advanced organizations developing, adopting, and utilizing the innovative artefacts in question [16]. The ADR process usually takes place in iterations over time with the stages of:

  1. 1.

    Problem formulation.

  2. 2.

    Building, intervention, and evaluation (BIE), usually in multiple cycles.

  3. 3.

    Reflection and learning.

  4. 4.

    Formalization of learning and outcomes [16].

The reported ADR process covers the time frame from 2016–2022, focusing mainly on Insta-Valmet co-operation, complemented with eventual other relevant consulting experiences by Insta of the subject matter.

2.1 Two Development Cycles: 2016–2019

Prior to this research during the 2010’s, such commercial concepts as BSIMM (Building Security in Maturity Model), OpenSAMM (Open Software Assurance Maturity Model), and OWASP (Open Worldwide Application Security Project) were discussed among the practitioners in Insta and VAS alike. These concepts focus on assessing the maturity and planning the adoption roadmap on a high-level, while providing limited practical support for adoption in R&D teams and no real-time visibility to adoption status. The first development cycle started in 2016. The goal was to develop an improved DevSecOps framework for VAS at R&D team-level and certified to comply with IEC 62443–4-1. In hindsight, this first iteration already resulted in several lessons towards the information-centric approach.

At first, the goal was simply how to get DevSecOps efficiently adopted in VAS. In 2018, after years of cybersecurity and DevSecOps consulting, the practitioner authors identified a more focused question: what kind of practice(s) would speed up the adoption of standard compliant DevSecOps among industrial suppliers while being repeatable and scalable so that new persons and teams can quickly learn to apply the practices.

After a successful audit in 2019, the practitioners noticed that the metrics and mechanism in terms of adoption status as well as model’s modularity and flexibility required improvement. This was the motivation for the second cycle in 2019, when the DevSecOps framework at VAS was refactored to be more modular and independent from the contextual needs of the initial R&D projects, and introduced the concept of an Internal Control, to make the adoption of the model measurable. The new model was evaluated to be successful in providing real-time visibility to the adoption status. During the evaluation at Insta, the practitioners identified an opportunity to build a separate tool that would make it easier to adopt DevSecOps in different organizations that might use different software engineering tools.

2.2 The Third, Fourth and Fifth BIE Cycles: The OXILATE Project 2020–2022

The third BIE cycle took place in from December 2019 to December 2020 when the OXILATE project started (https://itea4.org/project/oxilate.html) and a representative of a research organization joined the team. The development cycles from now on followed the ADR guidelines more consciously. Insta implemented a prototype of a dedicated “Dependability Tool” for managing the DevSecOps information model. During the Insta’s internal evaluation phase and with Insta’s customers, we (all the authors) realized that while a dedicated tool enabled improved automation and more convenient workflows, moving the management of the DevSecOps information model to a new separate tool may be challenging to adopt in practice.

In the fourth BIE cycle during the first half of 2021, Insta gathered information for “pivoting” the DevSecOps model of the third cycle and interviewed their current and potential customers about the business goals, challenges, and solutions in DevSecOps and Cybersecurity Management System adoption. The design principles and the two proposed design objectives presented in this paper are based on the evaluation of the fourth BIE cycle, and the fifth BIE cycle, which consisted of further development of VAS’ DevSecOps framework (and, at the same time, Insta’s reference framework) that took place during the second half of 2021 and the first half of 2022. This further development was motivated by retrospectives and end-user feedback, where we identified concrete improvement areas to simplify and clarify the information model. Formalization of learning through design principles.

The data documented in the consulting and BIE cycles consists of feedback from external auditors, meeting notes from retrospectives, formally documented continuous improvement reviews, and documentation of Insta’s customer interviews. The verbal interactions and sparring between Insta and VAS practitioners, and with Insta’s other customers over the years have also accumulated insight. This data has now been conceptualized as design principles and design objectives of this paper.

Sein et al. [16] suggest that the learnings from BIEs should be ultimately formalized as design principles, based on the accumulated experiences. Gregor et al. [5] suggested the generic form and components of design principles to include descriptions of implementers, their aims, the intended users, context, mechanisms, enactors, and rationales. Hence, our formalization of learning takes place through such descriptions of design principles (and design objectives for the emerging issues in the end of the last BIE).

3 Proposed Solution

Compliance with the IEC 62443–4-1 standard requires a documented development process with evidence of practicing the process. An IEC 62443–4-1 requirement generally begins with the phrase “A process shall be employed…”, after which the requirements state what must be done or what must not be done. In other words, the standard focuses on the tasks or procedures that the product supplier organization must employ.

The standard focuses on the actions that must be performed. However, it largely ignores the information artifacts that are related to the process, except as evidence to demonstrate that the processes have been practiced. The starting point for the proposed solution is the realization that the information artifacts, or the information model, also deserves attention: it is easier to understand a process if you consider both the procedures and the information model of the process, i.e., the conceptualization of information used and produced. The realization about the importance of the information model resembles Fred Brooks’ [2] famous remark about the relationship between code and data structures. We have modified the Brooks’ quote to support the proposed solution as follows: “Show me your process steps, and I shall continue to be mystified. Show me your process information model, and I won’t usually need your process steps.”

3.1 Adoption Challenges and Information Model

Adopting a DevSecOps process in practice in an R&D organization is not straightforward. Table 1 summarizes the challenges faced at VAS in the adoption of the DevSecOps process, and how each challenge relates to the information model.

Table 1. Challenges in DevSecOps adoption
Full size table

3.2 Design Principle 1: Information Model Before Process/task View

Table 2 formulates the basic realization of the proposed solution/method into a design principle of clarifying an information model for the process before detailing the process tasks or steps. In the context of VAS’ DevSecOps process, we instantiated design principle 1 with an information model that we call an issue graph. The artifacts of the process are called issues, which are linked together to form a graph. The issues may include, for example, process descriptions, project documentation, security-related issues, and internal controls to the content of the DevSecOps process.

Table 2. Design principle 1
Full size table
Fig. 1.
figure 1

An Entity Relationship Diagram of the “issue graph” process information model.

Full size image

Figure 1 presents the issue graph information model using the entity relationship diagram (based on the notation by [3]). Issues are identified uniquely due to the traceability requirements of the standard, and we maintain a modification audit trail. The issues of the same type follow the same workflow state machine, and each issue is in a specific workflow state.

The issues of the graph are generated by creating new branches to the graph from template branches. Each issue may include instructions or means for the user to instantiate new issues as children of the said issue. Each issue is owned by a user, who is responsible for maintaining this piece of content. Users can collaborate on the issues by commenting on them and by referring to them by a URL. Issues can be tagged or labelled to categorize them for different metrics and to facilitate the process. The issues can be linked together with different types of links.

Each of issue types has an issue-type-specific workflow state machine, which may have issue-type-specific custom fields. The issue types of the VAS [IEC 62443–4-1] process model include controlled documents for process descriptions and project documentation, security issues that need to be managed, internal controls, security requirements, tests cases and test executions (Table 3).

Table 3. Issue types of the [IEC 62443–4-1] process information model
Full size table

The main link type is a descendance relationship or parent-child relationship which produces a tree-based structure. The descendance hierarchy can be used for access control, by giving different users read or write access to different branches of the tree. In addition to the parent-child links, there are other types of links that capture the relationship between issues (e.g., a security requirement may mitigate a security issue). A test case may be designed to verify a security requirement or mitigation of a security issue. The types of most important issue links are described in Table 4.

Table 4. Types of issue links
Full size table

Common feedback received from developers and managers in the retrospectives over the years was that it is hard to understand what should be done in practice and concretely to follow the standard compliant DevSecOps process. At the same time, we found that it is not feasible to provide very specific step-by-step instructions, because they would be too long and tedious to use and maintain. To our experience, organizing the information related to the DevSecOps process more clearly has helped developers and managers to get an overview of the process and understand what needs to be done.

3.3 Design Principle 2: Information Model Modularity

Many of the benefits of the information model are based on its modularity, which we have described as a separate design principle in Table 5. When design principle 2 was instantiated at VAS, we designed the issue types so that each issue type has a workflow state according to an issue type-specific workflow state machine. This makes the state of the information model searchable and enables the creation of various metrics and statistics.

Table 5. Design principle 2
Full size table

We wrote small snippets of instructions, which we reused and included in both process descriptions and to the relevant contexts in various user interface views. This makes it easier to apply instructions that are relevant for the user in their current task. We also used a special issue type, internal control, which represents the standard DevSecOps tasks in a project. The internal controls can be labelled into separate adoption steps, so that a team can concentrate on a subset of the internal controls at a time. This helps with gradual adoption of the process.

The internal control is an issue type that models the standard tasks of the DevSecOps process. This concept is not included in the [IEC 62443–4-1] standard, and it is not used in all organizations that have a DevSecOps process. However, there are several benefits using internal controls: they help with gradual adoption of the process; they help with making the scoping decisions about which tasks are applicable and they enable a standard progress metric about the process adoption.

The workflow state machine of the internal control is shown in Fig. 2. The default state in the beginning is Open, which means that there is no decision whether the task is in scope for the project. Acceptable states are Not Required (task out of scope) and OK (task done). There is no end state, because DevSecOps is a continuous process. By tracking the last reviewed timestamp, controls in the OK/Not Required state can be highlighted to require attention. There is a state transition from the OK state to the same state, so that the last reviewed timestamp can be easily updated.

The notion of internal control was introduced in response to an R&D director’s request, at the end of the first BIE cycle, to make the adoption of the DevSecOps directly measurable with simple metrics. We have also tracked the adoption of the DevSecOps process by annual targets that we set based on metrics that we derived directly from the information artifacts: from the status of internal controls and security issues. To our experience, it is easier to lead the adoption, when R&D leaders can set measurable targets. Having a granular information model enables us to adopt the large standard compliant DevSecOpc process gradually and easily at team level.

Fig. 2.
figure 2

Workflow state machine for internal control

Full size image

3.4 Design Principle 3: Information Model Tailoring

In VAS, the DevSecOps process and thereby its information model needed to be tailored for the specific demands of the organization and in some cases even to the individual teams. The design principle of information model tailoring to the organization is presented in Table 6.

Table 6. Design principle 3
Full size table

The VAS’ DevSecOps process implements its issue graph information model mainly based on the Atlassian Confluence and Jira tools that have been an inspiration for many characteristics of the information model. These tools have limitations so that not all steps described here could be automated and, thus, have influenced the design of the hierarchy of the issue graph. Table 7 illustrates the Valmet Automation implementation.

Table 7. Implementation of the issue types of the issue graph model in VAS DevSecOps
Full size table

When we added new VAS teams to adopt the DevSecOps process after the first BIE cycle, we noticed immediately that the information model must be tailored team-wise and to fit the needs of projects of different sizes and different technology scopes.

As part of the Insta customer interviews during the fourth BIE cycle, we learned that the participants of the interviews preferred integrating security practices to their existing tools. The challenges of easily finding evidence of practicing a standard-compliant process are obvious to anyone who has had their process audited for certification. To our experience, it pays off to configure the tools that developers and managers already use so that evidence is accumulated automatically to the tools.

3.5 Design Objectives

Not all teams use the Atlassian tools for managing their work. Many use, for example, Azure DevOps, and Office for documentation. This justifies the design principle of information model tailoring, as a root cause for a lot of tedious work for DevSecOps process practitioners who try to support these teams. Documents and document templates must be converted between tool specific formats, and similar information needs to be maintained in multiple places. A portable representation format for the information model could help with these challenges, as a design objective for the future (Table 8). Our other design objective (Table 9) proposes to develop a process information model tool that would help with keeping the information model coherent across different tools. While there are existing tools for managing backlogs and tracking issues, tools for development documentation in an enterprise wiki, and tools for modeling the software architecture, the authors are not aware of any existing tools for managing the information model for a DevSecOps process.

Table 8. Design objective of portable information model representation
Full size table
Table 9. Design objective of process information model management tool
Full size table

4 Conclusions

The challenges of adopting DevSecOps and security standard compliance in agile OT development have been acknowledged in recent academic literature by Akbar et al. [1] and Rajapakse et al. [14] as well as in industry-oriented articles by Moyon et al. [10,11,12]. In this paper, we outlined one of the first experience reports on adopting and implementing IEC 62443 standard in practice. Based on the experiences, we proposed three design principles for standard compliant DevSecOps practices for OT development. These design principles originate from observations and experiences of several years in cybersecurity and software development practice in OT industry. The main influence on the creation of these design principles is that the same challenges or problems are encountered in numerous companies with minor variations. Altogether, the experiences suggest for an information-centric view on adopting and using the 62443–4-1 standard with DevSecOps to precede and complement the previously suggested process/task-centric view by Moyon et al. [10,11,12]. The information-centric view suggests that a shared information model of security issues gives common ground while allowing for more contextual, actual processes to integrate security work in DevOps. Such a common information model enables sharing, coordination, and reporting of security issues even when DevSecOps is implemented through often varying tasks across team-specific development processes and tools.

The information model behind the design principles emerged through the ADR BIE cycles to provide a theoretical background for the proposed solution. Design principles set up general guidelines on the adoption and implementation of IEC 62443, accelerating the operationalization of the standard into practice. As a case study, we described how the design principles are instantiated at VAS, while the formulation of the design principles suggests for their applicability beyond the case study at hand. Besides the design principles, we proposed two design objectives for the future: portable process information model representation and process information model management tool. These objectives are needed to address such technical questions as conversion between different formats and coherence of information model across tools.

This information-centric approach to adopt and implement complex standards such as IEC 62443 into practice complements the previously proposed process/activity-centric approach. Solutions in this paper are constructed abductively from empirical observations and development experiences to theory direction. The proposed approach sets up a new direction to the adoption and implementation of the requirements of IEC 62443 into practice and fulfils the hitherto addressed gap of missing experience reports in the scientific literature.