1 Introduction

Organizations face dynamic and rapidly changing environments that make the attainment of long-term economic success exceedingly difficult. The pressure to gain and retain a competitive advantage forces organizations to continuously identify means of cutting costs, improving quality, and reducing time to market (de Bruin et al. 2005). According to the resource-based view of organizations (Barney 1991), a competitive advantage can be secured by developing or acquiring valuable, unique, inimitable, and non-substitute resources that consist of assets and capabilities (Wade and Hulland 2004). While assets can be seen as the resource endowments of the organization, capabilities enable these assets to be deployed advantageously (Vorhies et al. 1999). For this reason, it is important for organizations to know their capabilities in depth. Maturity models have proven to be valuable tools that assist organizations in this endeavor (de Bruin et al. 2005). They support organizations in identifying and analyzing their capabilities to assess their overall maturity in specific domains. Maturity models are often applied enthusiastically because the insights that they can provide are highly valuable. However, after the initial application or, at the latest, when a targeted maturity level has been reached, the focus shifts to other organizational issues. This constitutes a problem, as organizations in all sectors are liable to change due to the complex and ever-changing technological, organizational, and economic environments in which they are embedded (Nelson and Winter 2004). Thus, capabilities can never be in a final state, as organizations are forced to adapt to this continuous change (Loasby 1998). This permanent transformation and the fact that maturity develops alongside capabilities, implying evolutionary progress, contradicts the notion of a singular and non-continuous maturity assessment (Mettler 2011). In the long run, the accuracy and relevance of maturity assessments can only be ensured by tracking capabilities and, therefore, by assessing current maturity levels over time. Moreover, most maturity assessment methods are specifically designed for certain maturity models and lack principles and phases that enable a generic application. For this reason, many models are tied to proprietary and limited assessment methods, such as questionnaires, which support their application. These methods may guide users during pure maturity assessment activities but eventually disclose how to effectively perform all relevant maturity model application activities (Mettler and Ballester 2021). Accordingly, there is a need for a generally applicable and continuous maturity assessment method (Englbrecht 2021; Frick et al. 2013; Stoiber and Schönig 2022). We define a continuous maturity assessment as an iterative and prolonged determination of the maturity level of an organization in a specific domain. In this context, the term continuous refers to a repeated maturity assessment that is conducted over specific intervals and which also considers individual changes in specific capability dimensions. Given these research gaps, we formulated the central research question (RQ) as follows:

RQ: How can any organization be guided through all phases of a continuous maturity assessment?

To answer the RQ and cover all its aspects adequately, we created the Continuous Maturity Assessment Method (CMAM). It supports organizations during all phases of continuous maturity assessments. The CMAM comprises five phases that can be implemented as an organizational routine covering all activities that are necessarily required to keep maturity assessments alive. We anchored the CMAM in the Design Science Research (DSR) methodology and designed it for practical use at all kinds of organizations. To ensure rigor, we followed the established method of Peffers et al. (2007) and complemented it with the Framework for Evaluation in Design Science Research (FEDS) by Venable et al. (2016). Our contribution is based on the inductive analysis of existing maturity assessment methods and considers best practices while addressing known inadequacies and weaknesses. These findings were used to attain the initially formulated design objectives, which are derived from the main RQ. A summative evaluation, including interviews with six market-leading organizations and an extensive case study provided valuable insights into the applicability of the artifact and its effectiveness.

The remainder of this article is structured as follows: in Sect. 2, we present the theoretical background of the research phenomenon under observation. Subsequently, in Sect. 3, we outline the underlying research methodology that we employed to create the CMAM. The CMAM is presented and explained at length in Sect. 4. In Sect. 5, we outline the results of the summative evaluations, which included a survey and an extensive case study. We discuss our contributions, the implications of the study, and its limitations in Sect. 6 and conclude with a summary of the findings in Sect. 7.

2 Theoretical Background

2.1 Organizational Capabilities, Maturity, and Maturity Models

Organizational capabilities can be defined as organizational entities that represent complex bundles of skills, accumulated knowledge, and systems that manifest in organizational processes (Kwon 2021). When deployed purposefully, capabilities enable organizations to perform certain activities to achieve particular goals and outcomes and serve as the fundamental basis of economic success (Kwon 2021). Maturity models have been used extensively to (i) assess the capabilities of an organization in a certain discipline, (ii) to provide a basis for benchmarking against competitors, and (iii) to guide an organization in the acquisition of the capabilities that it needs to improve in that discipline (Serral et al. 2020). In this context, maturity is a specific process that entails the explicit definition, management, measurement, and control of the evolutionary growth of an entity, such as - in this special case, capabilities (Kerpedzhiev et al. 2021; Paulk et al. 1993). Therefore, maturity implies evolutionary progress from an initial state to a final and more advanced one (Mettler 2011). A maturity model is generally structured as a sequence of distinct levels (Pöppelbuß and Röglinger 2011) that follow a path from an initial state of maturity to a potential final state of maturity (Becker et al. 2009). Those models are usually conceptualized as matrices, with maturity levels on one axis and capabilities on the other, while capabilities are mostly arranged along specific dimensions (Lasrado et al. 2015). Maturity levels are phases of development that are arranged sequentially from the lowest to the highest. Maturity models are particularly important for identifying the strengths and weaknesses of organizations by reference to an underlying phenomenon and for the collection of benchmarking information by the organization (Khoshgoftar and Osman 2009). Maturity models, as diagnostic or benchmarking tools, enable the identification of appropriate actions for creating or improving capabilities and, therefore, for reaching higher maturity levels (Kohlegger et al. 2009).

One of the first models was the Capability Maturity Model (CMM), which was designed to assess the maturity of software development processes (Paulk et al. 1993). Many models that followed the CMM were loosely based on it but lacked a comparable scheme. The capability maturity model integration (CMMI) project was initiated to create a standardized framework model (CMMI Product Team 2010). Although many models followed the basic outlines of CMMI, most of their authors did not disclose their research methods or the underlying design decisions (Mettler 2011).

2.2 Perspectives on Maturity Models

Research on maturity models can be viewed from two perspectives, representing cycles that include specific and sometimes overlapping activities. The developer perspective is directed at providing suitable and rigorously designed models, whereas the user perspective is oriented toward their effective and appropriate application (Mettler 2011; Proença et al. 2020). Both perspectives entail the use of different methods and frameworks for the creation, selection, and application of models. Figure 1 shows the two cycles and the associated generic activities.

Fig. 1
figure 1

Perspectives on maturity models according to Mettler (2011)

Full size image

2.2.1 Developer Perspective

The development of maturity models is highly complex and requires patterns to be recognized, structured, and documented so that the organization may improve its performance logically (Kühn et al. 2013; Röglinger et al. 2012). In the development cycle, the identified need or opportunity to develop a novel maturity model serves as a basis for defining its basic scope. This definition influences all parameters of decisions that are made during the design phase. To improve the model sustainably and iteratively, revision, in the form of evaluation, reflection, and appropriate adaptation, is necessary. These three phases, namely scope definition, design, and revision, have been recognized within maturity model research and are, in fact, essential for the developer perspective. Since many maturity models have been subjected to criticism because they are seen to oversimplify reality and lack an empirical foundation, research has approached the problem from a design process and design product perspective (Marx et al. 2012; Röglinger et al. 2012). In this regard, several procedural models have been developed to support the development of structured maturity models (Becker et al. 2009; de Bruin et al. 2005; Maier et al. 2012; Mettler 2011; Solli-Sæther and Gottschalk 2010; van Steenbergen et al. 2010). For example, de Bruin et al. (2005) investigated several maturity models in different domains and identified six distinct phases that guide the design of descriptive maturity models for prescriptive and comparative use (Röglinger et al. 2012). Another established method was proposed by Becker et al. (2009), who derived a procedure model from the well-known design science guidelines of Hevner et al. (2004). Overall, the developer perspective has a sufficient set of tools at hand to create rigorously developed and applicable maturity models. Especially due to the mentioned methods by de de Bruin et al. (2005) and Becker et al. (2009), the developer perspective has not been considered in depth within the study at hand.

2.2.2 User Perspective

As far as the application cycle is concerned, the need or opportunity for applying a maturity model must be determined. The intended use of a model may be descriptive, prescriptive, or comparative (de Bruin et al. 2005). This determination is followed by the laborious activity of identifying and selecting a model appropriate for the business. Once a model has been selected, a maturity assessment, in the narrow sense of that term, can be initiated. For most maturity models, a questionnaire supports the analysis of organizational capabilities. The findings of the assessment should then be taken as a basis for actions to improve or create capabilities and to reach a targeted level of maturity. Since the development cycle undoubtedly benefits from the introduction of development methods, potential users are often left to make essential decisions alone during the application cycle. Current assessment methods mainly address actual assessment activities (Mettler 2011). Depending on the internal resources at their disposal, organizations can choose between three approaches, namely self-assessment, third-party assessment, and complete outsourcing (de Bruin et al. 2005). Regardless of the chosen approach, numerous maturity assessment methods (also called “maturity appraisal methods”) have been developed. In contrast to the established methods of the developer perspective, no universally accepted maturity assessment method has been presented. Thus, contributions to effectively performing maturity assessments should be in focus within maturity model research.

2.3 Current State of Maturity Assessment Methods

Due to the popularity of maturity models, a potential user is confronted with various heterogenous maturity assessment methods that cover different phases of the application cycle. These include established methods like SCAMPI or ISO/IEC TS 33030, which are the de facto standard for a wide range of models and are updated and refined regularly by their managing organizations. Both methods cover maturity assessment activities and address, among others, the analysis of assessment results. Furthermore, numerous proprietary assessment methods have been developed. Those methods are tailored to particular maturity models and are not applicable generally. The existing maturity assessment methods can be classified along the dimensions of generality and scope. The term generality describes the degree of generic applicability to different maturity models. Methods can be highly specific, with concrete questionnaires, or relatively generic, with structures that are free of specific references to individual models. The scope of a method has to do with how the user is supported in all phases of the application cycle. Along both classification dimensions, the methods can be clustered into two disjunct areas (see Fig. 2). Area 1 comprises methods that have been developed for a specific maturity model and cannot be applied easily to others.

Fig. 2
figure 2

Classification of existing maturity assessment methods

Full size image

This area includes limited and highly specific questionnaires that support the user during the collection of evidence and information about capabilities (Akdil et al. 2018; Schumacher et al. 2016) or methods that also describe activities that support the user within subsequent actions (Adyrbai et al. 2021; Rosemann and de Bruin 2005). Area 2 comprises methods that are relatively generic and often more comprehensive. They are designed for well-established maturity model frameworks or sets of related models within a specific domain. The most popular methods of Area 2 are SCAMPI and the ISO/IEC TS 33030. SCAMPI was originally designed for evaluating organizations against the CMMI process model, but the procedure can also be used for a limited number of related process models, such as ISO/IEC 12207 or DIN EN ISO 9001. The ISO/IEC TS 33030 is a revised version of the ISO/IEC 15504, the former Software Process Improvement and Capability Determination (SPICE), which guides users in the assessment of organizational processes and capabilities. Another contribution to Area 2 is a framework defined by Mettler (2011), which describes activities that are relevant for users as well as important decision parameters that they must consider. The 360 Digital Maturity Assessment (360DMA) is a less generic method. It is mainly applied to a range of models for digitalization but also outlines general principles that may fit any maturity model application (Colli et al. 2019). Finally, the Maturity Model Architect (MMArch) by Proença and Borbinha (2018) supports the execution of maturity assessments by using enterprise architecture models, ontologies, and description logics.

2.4 Inadequacies of Existing Maturity Assessment Methods

Despite a large number of existing maturity assessment methods, there are some inadequacies and weaknesses that call for further research. Some of these weaknesses have already been identified in the literature, while others can be isolated by investigating and analyzing the most relevant methods and the activities they include.

2.4.1 Irreconcilability of Generality and Comprehensiveness

The first weakness is the apparent incompatibility of generality and comprehensiveness within established maturity assessment methods that belong to Area 2. In this sense, the proprietary methods from Area 1 are irrelevant because, for the most part, they are neither generic nor particularly comprehensive. Most existing comprehensive assessment methods are tailored to specific maturity models and are therefore not generally applicable (Tarhan et al. 2016). This is true for methods such as SCAMPI or ISO/IEC TS 33030. In this regard, SCAMPI was explicitly developed for CMMI for Development (CMMI-DEV), CMMI for Acquisition (CMMI-ACQ), CMMI for Services (CMMI-SVC), and other CMMI derivatives. The other established method, ISO/IEC TS 33030, is only applicable to SPICE for Software Development (ISO/IEC 15504-5), Automotive SPICE, and SPICE for System Development (ISO/IEC TR 15504-6). Both methods include detailed activities that prevent their application to other models. In principle, there is no generally accepted comprehensive assessment methodology (Frick et al. 2013; Mettler et al. 2010).

At the same time, more generic assessment methods, such as the 360DMA or the framework proposed by Mettler (2011), do not include comprehensive and detailed activities that support users during all assessment phases. They often lack activities that support the collection of capability data and the accurate reporting of assessment results. Moreover, they lack principles that would enable the method to be embedded into an organizational routine and the generated results to be used for action plans. This is important as assessment methodologies should harness the knowledge that is generated from their application (Rosemann and Vessey 2008). In general, there are few rigorous assessment methodologies (Frick et al. 2013) that are both general and comprehensive. A new method must therefore be generically applicable to all maturity models and contain activities and principles that cover all assessment phases.

2.4.2 Lack of Continuity

Existing maturity assessment methods are insufficiently sensitive to the problem of continuity (Proença and Borbinha 2018). Most methods do not require iterative assessments specifically or indicate that such assessments should only be performed over long intervals. SCAMPI assessments, for example, are performed every three years, which is a long period given the rapidly changing business environments (Albuquerque et al. 2019). Moreover, some studies report difficulties with continuous assessments at different organizations (Fontana et al. 2018; Uskarcı and Demirörs 2017). For existing methods, no mechanism is in place for enforcing the general continuity of application (Uskarcı and Demirörs 2017). Existing assessment methods only focus on the collection of evidence to substantiate maturity level calculations without highlighting the importance of a continuous procedure (Proença and Borbinha 2018). However, continuity is critical because capabilities are liable to change, either due to organizational improvements and capability creation or due to deterioration (Loasby 1998). The adoption of a structured and continuous maturity assessment routine is a prerequisite for the effective development and maintenance of knowledge about organizational capabilities. By introducing such routines, maturity models can describe how organizational capabilities develop over time (de Bruin et al. 2005) while evaluating and promoting their continuous improvement (Bititci et al. 2015).

3 Methodology

3.1 General Research Approach

Given the existing inadequacies and weaknesses, we aimed to develop a novel maturity assessment method combining generality, comprehensiveness, and continuity. In line with existing research on maturity models, we positioned our contribution within the DSR paradigm. This decision enabled us to adopt established principles and draw on methodological guidance for the development of the artifact and its evaluation (Peffers et al. 2007). We relied on the well-established process model of Peffers et al. (2007), which is based on the methodology of Hevner et al. (2004) and provides a detailed development process for conducting DSR on information systems. The process model translates the guidelines and DSR principles of Hevner et al. (2004) into an easily applicable process. This translation enables DSR endeavors to proceed in a straightforward manner. Based on the inadequacies and weaknesses of the existing maturity assessment methods, which we described in Sect. 2.4, we defined four distinct objectives. Subsequently, we executed two development phases that included semi-structured interviews as formative evaluations. These interviews helped us to estimate and evaluate the compliance of the artifact with the design objectives (Stefanou 2001). Once the two development phases had concluded, we conducted two summative evaluations to produce empirically based feedback.

First, the artifact was assessed by practitioners from different domains to elicit interpretations and feedback from different practical contexts (Venable et al. 2016). Second, an extensive case study of a real-world setting was conducted over six months. All evaluations were conducted as part of a comprehensive evaluation strategy in line with the FEDS (Venable et al. 2016). Figure 3 overviews the development and evaluation phases.

Fig. 3
figure 3

Development and evaluation phases

Full size image

3.2 Definition of Design Objectives

Four concrete design objectives could be deduced from the RQ and from the inadequacies and weaknesses of the existing maturity assessment methods that we identified. Those objectives were used for orientation and guidance during the design and development phases. The first design objective (DO1) is that the design and the structure of the artifact must be understandable and easy to apply for practitioners. The second design objective (DO2) reflects the need for principles and activities that enable a continuous maturity assessment. The artifact must refer to activities that entail the tracking of capabilities and thus create a basis for an iterative assessment. The third and fourth design objectives refer directly to the classification of the artifact within the set of existing assessment methods, as described in Sect. 2.3. First, the artifact should be applicable to the largest possible number of maturity models and organizations without having specific links to existing models, that is, it should possess a high degree of generality (DO3). To that end, it is useful to define a generic assessment method that makes use of relevant information when applied at specific organizations. What information is relevant depends on the actual use case. Furthermore, the artifact should provide comprehensive activities that support users during all phases of the maturity assessment (DO4).

3.3 First Development Phase – Structuring What Exists

In the first development phase, we aimed to create an empirical basis, gain insights, and synthesize findings and best practices from past and current research on maturity assessment methods. To identify an appropriate selection of literature, we performed two structured literature reviews (SLR) to investigate both areas that we outlined in Sect. 2.3. These SLRs helped us to create a theoretical foundation for the development of our artifact (Sturm and Sunyaev 2019). The first SLR concerns the methods of Area 1 and thus investigates maturity models that are accompanied by proprietary assessment methods. The second SLR focuses on the assessment methods of Area 2.

We performed both SLRs according to the structured method of vom Brocke et al. (2009) and considered the most relevant journals and conference proceedings in the research domain by querying the databases ACM Direct Library, AISeL, IEEE Xplore, ScienceDirect, Scopus, and Springer Link. A detailed overview of the SLRs, including the search strings, the eligibility criteria, and all analyzed articles can be found in Appendix 1 (available online via http://link.springer.com). During the SLRs, we found 45 eligible articles in Area 1 and eight articles in Area 2, which we analyzed in detail. Subsequently, we extracted the relevant data by using grounded theory, a qualitative research method that seeks to develop a theory that is grounded in data that is systematically gathered and analyzed (Urquhart et al. 2010). We aimed to identify patterns, common attributes and principles, and best practices from existing maturity assessment methods that can be used as a basis for an artifact. Accordingly, we applied the methods of open and axial coding, as proposed by Strauss (1997). In the first round of coding, each of the authors analyzed 20 publications from the sample. Open coding, an interpretative method, was used to disaggregate all assessment methods into their constituent parts analytically. The goal was to develop substantiated codes that would enable those parts to be described, named, and classified. The breakdown of each assessment method into separate activities is an example of this open coding approach. Codes were then assigned to the activities. After this first round of coding, we collated, compared, and contrasted the codes that we had identified. Eventually, we harmonized the individual interpretations of the main codes. In the second round, we applied the method of axial coding to connect the formulated codes to each other. During this process, we organized the codes from the previous round into categories. This categorization enabled the creation of phases within the maturity assessment methods that comprise similar activities. After a second discussion, the results were harmonized again. In round three, the remaining publications were coded with the findings of rounds 1 and 2 to test them against data. Subsequently, we clarified and resolved any remaining coding differences. Following inductive reasoning, as suggested by Hempel (1966), we extensively discussed the created codes and categories to identify best practices and the fundamental principles of maturity assessment. Table 1 shows the identified categories and codes, which are translated into phases and activities that are essential for maturity assessment methods. The table also overviews the descriptive statistics within the underlying articles. In total, we derived five generic phases, which cover all activities that form part of the investigated methods. A central finding that emerged from the coding is that few methods address the selection of models. Moreover, most proprietary methods in Area 1 only cover the preparation, assessment execution, and reporting phases. They do not provide support for adaptations or critical reflection on assessment results. Figure 4 shows how the identified phases can be mapped onto common maturity assessment methods of Area 1. After the SLRs and coding activities, we conducted seven semi-structured interviews with researchers and practitioners. This included two professors, one postdoctoral researcher, and four management consultants completing their doctoral degrees. Those interviews would serve as a formative evaluation episode EV1.

Table 1 Status quo analysis of existing maturity assessment methods
Full size table
Fig. 4
figure 4

Phases and activities in established maturity assessment methods

Full size image

All experts possessed considerable expertise in the field of maturity models, whereas the consultants had already been involved in their practical selection, implementation, and monitoring at different organizations. The experts were presented with the RQ, the design objectives, and the created findings on maturity assessments. Against this background, they were asked which fundamental phases, activities, and indispensable aspects of maturity assessments should be considered for artifact creation. Moreover, they were asked to indicate what extensions would be necessary to achieve the design objectives. Appendix 2 shows the details and the results of the interviews.

3.4 Second Development Phase – Advancing What Exists

The analysis of existing maturity assessment methods made it possible to overview the status quo and to identify fundamental phases, activities, and principles. Furthermore, the results of the formative evaluation EV1 provided us with expert knowledge on the necessary and potential extensions that would make the design objectives easier to attain. This expert knowledge formed the basis of the second development phase, in which we created the final CMAM by extending and advancing existing knowledge about assessment methods. To that end, we followed the method of abductive reasoning, a creative process that enables the introduction of new concepts and ideas (Peirce et al. 1998). Abduction can extend and create knowledge because researchers imagine and analyze all possible theoretical accounts of a given problem or a set of data and then form hypotheses until they arrive at the most plausible interpretation (Charmaz 2008). We built the CMAM iteratively by selecting the most useful phases and activities from the first development phase and extended it by incorporating information from the formative evaluation EV1. After each iteration, we performed semi-structured interviews (EV2) with the expert panel to obtain additional feedback. After three iterations and 21 interviews, we finished the procedure because the experts did not identify further extensions or new issues.

4 Results

The CMAM that we developed reflects best practices and contains elements that are indispensable to an appropriate maturity assessment method. In addition, it extends and advances prior methods to address the formulated design objectives. The CMAM is intended to guide organizations through all the steps of the maturity model application, and its phases can be implemented into an organizational routine.

The CMAM consists of five phases and several activities that are arranged iteratively. Figure 5 shows all phases and contains descriptions of the associated activities, their purposes, and their objects. The first phase, maturity model preparation, is the user’s point of entry into the assessment method and contains activities that revolve around the selection of an appropriate maturity model and preparation for its use. It is followed by the phase assessment specification instantiation, in which all relevant responsibilities, decision parameters, and assessment details of the method are defined. The users must instantiate a metamodel that results in an assessment specification that is unique to the individual assessment. In the next phase, capability tracking and assessment, all relevant capabilities are tracked by collecting associated data in line with a trigger- or interval-based schedule. The resulting data is used to define the maturity level, which is then communicated and analyzed further in the reporting and action plan phase. Finally, the maintenance or termination phase describes activities that are related to the adaptation or termination of the procedure.

Fig. 5
figure 5

The continuous maturity assessment method (CMAM)

Full size image

4.1 Phase 1 – Maturity Model Preparation

The first CMAM phase lays the foundations for all subsequent ones by having the users select and prepare an appropriate maturity model and formulate basic assessment definitions. For a suitable model to be selected, several decision parameters must be considered. They depend on business needs and situational factors. These factors and needs can include the origin of the model (i.e., academia or practice), its reliability (i.e., evaluation), its accessibility (i.e., cost of use, if any), its practicality (i.e., whether it is problem-specific or more general), and its design mutability (i.e., the convertibility of the model and the ease with which it may be integrated into the existing organizational model base) (Mettler 2011). Furthermore, the assessment approach is highly relevant. For example, much depends on whether the organization can perform all activities as self-assessments or if external support or the retention of certified experts is necessary. The latter can be relevant to assessments that are performed as part of contractor evaluations (Paulk et al. 1993). In addition, the user should become aware of the added value of the continuous application and of the resources and capacities that are required. This awareness depends strongly on the anticipated assessment goals, such as benchmarking against competitors or creating knowledge about internal capabilities (Serral et al. 2020). Thus, creating awareness necessitates a conclusive resolution of uncertainties or ambiguities in the assessment goals (SCAMPI Upgrade Team 2006). If the selected maturity model cannot be adopted because of the organizational situation and environment, goal-oriented adaptation and adjustment are possible. However, these should not change the fundamental structure of the model or contradict its objectives. Another important prerequisite to enabling an effective CMAM application is the creation of awareness about the relevance and importance of the model. This exercise should also highlight the need for continuity (Stoiber and Schönig 2022). All involved users must understand that continuous assessment is relevant and crucial to arriving at accurate and objective insights into organizational capabilities. Moreover, for the CMAM to be embedded into an organizational routine sustainably, all stakeholders must guarantee their commitment and long-term dedication (Colli et al. 2019).

4.2 Phase 2 – Instantiation of Assessment Specification

Once an appropriate maturity model has been selected and all activities in Phase 1 have been performed, assessment details must be specified to enable the introduction of a systematic and iterative organizational routine. To that end, an assessment specification is created. It includes all vital building blocks, such as responsibilities, a data setup, and conditions for maintaining or terminating the CMAM. In this sense, the assessment specification describes objects, parameters, and characteristics of the real-life application of the maturity model. To facilitate the enumeration of substantiated assessment details, the CMAM provides a metamodel that can be used to instantiate the assessment specification for any underlying maturity model. Figure 6 shows the metamodel, which is presented as a Process Data Diagram (PDD), an approach that includes standards of the Unified Modeling Language (UML) (van de Weerd and Brinkkemper 2008). The PDD has already been used in related research and is sufficiently expressive for creating an appropriate model (van Steenbergen et al. 2010). The process view on the left-hand side of the diagram is based on a UML activity diagram, and the deliverables view on the right-hand side is based on a UML class diagram. The user performs the process on the left side to specify the classes and attributes on the right side. These classes represent all decision parameters and details that are relevant to the subsequent phases of the CMAM. Due to its representation as a class diagram, the metamodel can be implemented in different ways, for example, as a manual routine or as software that includes a database for defined and collected data.

Fig. 6
figure 6

Assessment specification metamodel

Full size image

4.2.1 Data Setup

In the first phase, data setup, the required data, its sources, and its processing must be specified for the purposes of the maturity assessment. This phase refers to similar data collection and preparation activities in SCAMPI and ISO/IEC TS 33030. At first, a product owner of the CMAM must be nominated to create responsibilities. Then, capability dimensions are selected iteratively, and the corresponding data types, which are used to identify capabilities, are defined. A qualitative acquisition principle must be specified for qualitative data. This principle also refers to participant groups that gather relevant data. In most proprietary assessment methods, data is collected by using questionnaires. If a quantitative data-acquisition principle is adopted, the data source for the capability dimension must be specified. These sources may include the databases of Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), or Business Process Management (BPM) systems. For instance, as far as the supply chain management maturity model of Lockamy and McCormack (2004) is concerned, data for days of supply (DOS) or cash-to-cash cycle times, which is easily accessible, could be used to define and assess relevant capabilities. Since most maturity models contain qualitative descriptions of capabilities, individual metrics can facilitate the translation of the acquired data into specific capabilities or capability levels. Moreover, similarly to SCAMPI, data validation is necessary to ensure that the assessment is reasonable and accurate.

4.2.2 Tracking Setup

In the second process phase, tracking setup, the details of capability tracking, and the assessment must be defined. Thereafter, the capability tracking schedule is set. It can be based on events or intervals. In the case of event-based schedules, specific triggers must be identified. Those triggers should indicate that specific capability dimensions ought to be assessed. For time-based schedules, the relevant dimensions are examined over fixed intervals. Subsequently, the capability dimensions must be selected that should be assessed for the set intervals and triggers. For a full scope, all capability dimensions of the maturity model, and therefore all individual capabilities, are tracked by reference to the corresponding schedules. For an individual scope, specific capability dimensions can be selected that correspond to a defined schedule.

4.2.3 Revision Setup

To be applicable over a long period, the CMAM should contain principles that allow for flexible adaptations of its setup. Furthermore, being an iterative procedure, it must have clear termination conditions. Both principles are defined in the revision setup stage, in which maintenance and termination conditions are specified. Any changes in responsibilities or data sources need to be adapted to the assessment specification. Moreover, changes in the management of the organization or its structure may lead to the termination of the CMAM. Finally, the introduction of superior maturity models may require the CMAM to be terminated and restarted because Phase 1 is always the entry point of new models.

4.3 Phase 3 – Capability Tracking and Assessment

In the third phase of the CMAM, the predefined assessment specification is used to track all relevant capabilities according to the formulated schedules. The product owner is responsible for either enabling the collection of quantitative data from the data sources and translating it into capabilities or for gathering capability information through qualitative methods, such as questionnaires and focus groups. The tracking results are then used to define the current maturity level. Tracking is also related to the RQ in that it enables the creation of a method for the continuous assessment of maturity through the constant monitoring of organizational capabilities. The information that is obtained thus can be used to determine maturity levels. Stakeholders obtain a more detailed and recent representation of the state of affairs, and the needs of many maturity model users are met, especially in highly dynamic environments (Englbrecht 2021; Stoiber and Schönig 2022).

4.4 Phase 4 – Reporting and Action Plan

In the fourth phase of the CMAM, the results of the maturity assessment are reported to the stakeholders and other focal groups. While it is not described in detail in many existing methods, reporting is an important activity in the SCAMPI, ISO/IEC TS 33030, and 360DMA, as well as in other methods. An appropriate debriefing includes a presentation of results and a comparison with prior assessments, which enable learning and the formulation of action plans. The results must be analyzed critically, and it is necessary to decide whether advances in maturity should be coupled or uncoupled from the regular target system of the organization and if improvement-related activities can be conducted ad hoc or whether specific project initiatives are necessary (Mettler 2011).

4.5 Phase 5 – Maintenance and Termination

In the last phase, the user performs activities to ensure a valid CMAM representation, including an actual and correct assessment specification. Since organizational changes may necessitate the adjustment or adaptation of the specification, the user can decide to return to Phase 2 and rerun the assessment specification instantiation. The necessity of this operation depends on the maintenance conditions from the assessment specification. These conditions could include changes in organizational structures, stakeholders, or information systems for data collection. In some cases, the termination of the CMAM might be appropriate or necessary. The termination conditions may include, among others, the achievement of maturity goals, significant changes within the organization, or the introduction of a new and more suitable maturity model. In general, this phase ensures that the CMAM is validated and updated continuously, and it eventually results in the rationally justified termination of the assessment. If no maintenance or termination condition is met within a given iteration, the user is guided back to Phase 3 to track capabilities and to keep the maturity assessment alive.

5 Evaluation

5.1 Evaluation Setup

Accurate evaluation is a central and critical part of DSR (March and Smith 1995), and its aim is to assess the utility that an artifact contributes to its environment (relevance cycle) and the knowledge that it adds to the knowledge base (rigor cycle) (Hevner et al. 2004). Therefore, we followed the FEDS of Venable et al. (2016), which complements the process model of Peffers et al. (2007) and extends it by introducing detailed evaluation principles. The FEDS complements and details the generic evaluation phase of Peffers et al. (2007) by introducing tools to create an overarching evaluation strategy. In this regard, it supports the (i) explication of evaluation goals, (ii) the development of an appropriate evaluation strategy, (iii) the determination of evaluation properties, and (iv) the design of evaluation episodes. The main goal of the evaluation was to support the achievement of the design objectives (ex-ante) and, eventually, to measure and assess the degree of attainment (ex-post). These two goals mean that a combination of formative and summative evaluations is required. We designed four evaluation episodes, two of which are formative (EV1 and EV2) and two of which are summative (EV3 and EV4). Table 2 presents an overview of the evaluation episodes, including the three guiding questions of evaluation in DSR, namely “why?”, “how?” and “what?” (Prat et al. 2015). The formative evaluations EV1 and EV2 were used to produce empirically based interpretations that provided a basis for improving the characteristics and the performance of the CMAM (Wiliam and Black 1996). For the summative evaluations EV3 and EV4, we defined four evaluation criteria that allowed us to conclude the attainment of the evaluation goals.

Table 2 Performed and planned evaluation episodes
Full size table

First, we chose the criterion of operational feasibility, which concerns the degree to which managers, employees, and other stakeholders might support the proposed artifact effectively, operate it, and integrate it into their daily practices (Mark et al. 2007). This process is essential because only feasible artifacts can be applied and maintained by organizations. Secondly, we evaluated the generality of the CMAM to ensure that the artifact can be used at any organization and for any maturity model. Third, the criterion of applicability was selected. Fourth, we evaluated the effectiveness of the CMAM, which we defined as the degree to which the artifact achieves its goal in real-life situations (Prat et al. 2015).

5.2 Expert Survey

To assess the operational feasibility and generality of the CMAM, we conducted expert surveys at six organizations. To cover different scenarios and to collect heterogeneous feedback, we selected organizations from different industry sectors, of different sizes, and with varying experiences of maturity models. The organizations in question included market-leading businesses in the chemical industry, plant engineering, the manufacturing sector, and financial services. At least one interviewee at each organization was responsible for applying a maturity model at a specific business unit. In Step 1 of the interviews, the underlying RQ, the design objectives, and the CMAM, including all its phases and activities, were presented in detail. In Step 2, we proceeded with initial questions that were aimed at gathering information about the position of the interviewee within the organization, their experience with maturity models, and their awareness of the RQ. Subsequently, in Step 3, we administered a questionnaire. It contained 12 statements, and the interviewees were asked to indicate their agreement or disagreement with each. The questionnaire used the well-established Likert scale (Likert 1932), which has interviewees record their level of agreement or disagreement with a statement on a symmetric agree-disagree scale. The statements were formulated in a way that enabled us to draw direct conclusions about the two formulated evaluation criteria. In the final step, Step 4, we conducted a semi-structured interview that helped “to confirm what is already known whilst at the same time providing the opportunity for learning” (Kundisch et al. 2021; Recker 2013). Appendix 3 provides a comprehensive overview of the survey details and the results. The survey showed that all organizations were aware of the relevance of continuous maturity assessment and would implement principles and methods that support it. They pointed out that the phases and activities of the CMAM could be adopted as organizational routines within different departments and for different maturity models. The survey results also demonstrated that managers would support the adoption of the CMAM and that they saw the considerable potential benefits of its application.

5.3 Case Study

After the first summative evaluation yielded positive feedback on the operational feasibility and the generality of the CMAM, the artifact was evaluated further in an industrial case study. The case study had the distinct objective of testing the hypothesis that the CMAM is an effective and applicable method for continuously assessing organizational capabilities and maturity.

5.3.1 Case Study Design

Since the CMAM constitutes an iterative process that is applied over time, the case study was designed as a longitudinal study that is sensitive to temporal variations. To that end, the CMAM was introduced at an organization to enable the application of a maturity model over a period of six months. The organization in question already had experience with applying such models, which enhanced the significance of the study. All steps of the application of the CMAM were documented and analyzed. This process covered the organizational implementation of the CMAM and the operational results of each phase. After an application period of six months, the users at the organization were asked for feedback through a survey and during semi-structured interviews. The aim was to determine whether the CMAM is (i) applicable and implementable as an organizational routine, and (ii) an effective method that enables continuous and comprehensive maturity assessments.

5.3.2 Case Study Setup

While all organizations that participated in evaluation episode EV3 agreed to participate in a case study in principle, two of them initiated concrete action after the survey was conducted. Ultimately, the chemical organization was selected for extensive study because it agreed to the documentation of all internal data. The case study was conducted over six months in 2022, between February and August. In the beginning, a project team was set up. It consisted of four members of the organization and the authors. The CMAM was introduced at the IT Service Management department of the organization, which was responsible for delivering business applications and IT-enabled processes at the German headquarters of the business. Since 2015, the department has been using an updated version of the original Gartner Infrastructure Maturity Model (GIMM) (Hidas 2006). A detailed overview of the updated GIMM from 2015 may be consulted in Appendix 4. Previously, maturity had been assessed every two years as part of a self-assessment and with limited external support. This self-assessment was conducted based on the descriptions of the GIMM and loosely followed the generic phases of SCAMPI. External consultants had created questionnaires that were intended to enable conclusions to be drawn about the capabilities of the GIMM. The questionnaires would be distributed within the department. At the last assessment, which took place in 2021, a maturity level of “Rationalized” had been achieved, which represented no improvement on the previous assessment, which had taken place in 2017.

5.3.3 Implementation of the CMAM as an Organizational Routine

Initially, all relevant phases and activities of the CMAM were discussed with the project team. Subsequently, the first phase, maturity model preparation, was executed by the organization-side project team. Despite the last update of the maturity model in 2015, the project team decided to retain the GIMM for the case study because its capabilities were still relevant to the department's goals. However, the project team included a termination indicator that would necessitate the selection of a new model or an updated GIMM if the management demanded a new strategic alignment. The detailed results of all activities are presented in Appendix 5. The second CMAM phase was performed thereafter. The assessment specification was instantiated and visualized in MS Visio. Once the product owner had been defined, the data setup was formulated according to the PDD. Data sources that indicated where to gather information about a given capability were identified for each capability dimension. This process led to the definition of acquisition methods for the dimensions in question.

The data for four capability dimensions was to be gathered through qualitative questionnaires, while quantitative data for the two remaining dimensions would be extracted from databases. Then, translation metrics were created for each capability dimension, which enabled the questionnaire results and the data from the databases to be translated into distinct capabilities. Participant groups were identified for each questionnaire, and schedules were defined within the tracking setup. These schedules included one general interval, whereby a full assessment would be initiated every six months. Two separate automated triggers were determined for the two capability dimensions that were associated with quantitative data acquisition methods. Finally, three maintenance and termination conditions were set. Since the visualization of the assessment specification that was created included structured assessment information, a basic dashboard could be developed. Figure 7 shows the dashboard’s home screen, which was programmed by the department and was mainly used for enhanced visualization. The dashboard includes five primary tabs that overview all details of the assessment specification. It can also disseminate alerts if database queries indicate that there have been changes within the two quantitative capability dimensions, and it displays a timer for the fixed six-month interval.

Fig. 7
figure 7

CMAM dashboard

Full size image

5.3.4 Longitudinal Study

The product manager used the dashboard for six months to execute CMAM Phase 3, Phase 4, and Phase 5 iteratively. Two automated database triggers were activated during the longitudinal study, and one fixed six-month interval passed. The capability dimensions that corresponded to the two database triggers were tracked. This tracking resulted in the identification of capability improvements, which led the department to reach a new and higher maturity level. In accordance with CMAM Phase 4, this change was reported to all stakeholders, and a comparison between the set goals and future actions was discussed. Due to the expiry of the fixed interval after six months, the product manager had to distribute questionnaires to all participant groups. However, the analysis of the capability dimensions did not lead to any further changes. At no point was any maintenance of the assessment specification required, and the CMAM was not terminated. Appendix 6 presents all events that occurred during the six-month study.

5.3.5 Study Results

The implementation of the CMAM and the six-month study allowed us to draw valuable conclusions about the applicability and effectiveness of the method. At the end of the study, we performed a survey and conducted interviews with the project team to collect evidence of the achievement of the evaluation objectives. We followed the procedure that was described in Sect. 5.2. First, we performed a survey. The project team members could indicate their agreement or disagreement with statements on a Likert scale. Subsequently, we asked each team member open questions to collect individual feedback on the applicability and effectiveness of the CMAM. Appendix 7 contains a detailed overview of the questions and the statements.

The project team members stated that, in principle, the CMAM “[…] was generic enough to be applicable to our GIMM and department environment.” The generic and instantiable assessment specification, in particular, solved the problems that had been generated using other methods in the past. Those methods had been too specific and did not match the GIMM. The statements of the project team members confirmed the applicability of the method as an organizational routine. Furthermore, the project team confirmed the effectiveness of the CMAM by indicating that it is comprehensive and continuous. The activities pertained to CMAM Phase 1 made it possible to “critically question the existing maturity model” and raise the necessary awareness among all stakeholders. In particular, the appointment of a product manager anchored CMAM as a routine process and gave it additional relevance. The team also stated that “the clear assessment specification of CMAM Phase 2" was particularly useful, enabling the visualization of all necessary assessment details. No such structured presentation had been developed previously. The specification also made it possible to develop a software representation because the UML standard of the PDD provided all the necessary information. Moreover, the two quantitative triggers allowed new perspectives to emerge from the maturity assessment. So far, only qualitative questionnaires have been used as bases for capability assessments. The introduction of these triggers and an additional fixed interval enabled changes in maturity level to be detected two months into the study. Under the previous regular assessment method, achieving the same outcome would have taken nearly two years. This circumstance made it possible to “update stakeholders about the achievements at an early stage and to adjust further goals.” These statements demonstrate the CMAM’s effective principles for continuity, as during six months, two changes within the capabilities and a major maturity level change could be identified. The project team decided to retain the CMAM as an organizational routine because it has measurable advantages over prior methods and because it is conducive to long-time usage, independently of the chosen maturity model.

6 Discussion

6.1 Contributions and Differentiation from Existing Methods

This article proposes the CMAM, a comprehensive method for the continuous tracking of capabilities and the assessment of maturity that can be applied to any model and at any organization. To develop it, we analyzed the existing literature on maturity models, identified inadequacies and weaknesses, and formulated a central RQ. Drawing on the four design objectives that we defined, we extended and improved existing assessment methods on the dimensions of generality, comprehensiveness, and continuity.

6.1.1 Managing Generality and Comprehensiveness

As noted in Sect. 2.4, there is a discrepancy between generically applicable and comprehensive maturity assessment methods. The CMAM addresses this issue and provides generality and comprehensiveness across its activities and phases. First, the CMAM does not refer to specific models and therefore allows for generic and configurable applications. The generic assessment specification metamodel provides means for individual instantiations. The CMAM improves on methods such as SCAMPI and ISO/IEC TS 33030 that are only applicable to a specific set of models. The evaluation episode EV3 showed that the CMAM could be adopted at all the organizations that we interviewed and used to apply the maturity models that they use. Furthermore, the case study confirmed that the CMAM is easy to configure and instantiate.

The CMAM is also comprehensive. It adopts phases that are established in previous methods, such as assessment preparation, assessment execution, and reporting, and it adds features that provide more comprehensive guidance. Phase 1 begins with the selection of an appropriate maturity model, the definition of a product manager, and the creation of sustainable awareness among stakeholders. This is not the case under SCAMPI, ISO/IEC TS 33030, or 360DMA. Furthermore, the definition of maintenance and termination criteria in CMAM Phase 5 is an improvement on existing methods because it introduces activities that ensure sustainable implementation. The case study that we conducted showed that the extended Phase 1 and Phase 5 increased awareness and relevance as well as engagement with the maturity model that is in use. Furthermore, creating a detailed assessment specification (CMAM Phase 2) is not a feature of any previous model. The assessment details can be mapped concretely by defining tracking intervals, automated triggers, and a fundamental data basis. As noted in the description of the case study, it was even possible to develop a digital representation and to track two dimensions automatically.

6.1.2 Enabling Continuity

Another design objective was to enable continuous capability tracking and maturity assessments. The final CMAM maps this through the iterative Phase 3, Phase 4, and Phase 5, as well as through the assessment specification. Previous assessment methods do not permit the continuous tracking of capabilities and do not recommend maturity assessments based on empirical data. For example, SCAMPI only specifies that the assessment must be repeated at least every three years. Far-reaching changes may occur over such a period, especially in sectors that are characterized by frequent innovation. Accordingly, the CMAM enables assessment schedules to be formulated. These schedules can include fixed intervals or triggers that are based on changes within the organization. Improvements and deteriorations can be detected rapidly, and countermeasures may be taken if necessary. The case study showed that changes in organizational capabilities and maturity levels are detected much more rapidly through the CMAM and can be used to adapt goals and action plans.

6.2 Theoretical and Practical Implications

Given the novel aspects of the CMAM, the implications of our research are twofold. As far as theory is concerned, the present paper analyzed existing assessment methods, investigated weaknesses and research gaps, and described a novel DSR artifact. It developed the first systematic analysis of existing assessment methods against the background of comprehensiveness, generality, and continuity. In this way, the analysis of the literature and the existing maturity models identified new perspectives and addressed existing research gaps concretely. The inclusion of continuity as an essential aspect of maturity assessments reflects a new perspective on the manner in which maturity must be assessed within rapidly changing business environments. The CMAM is a new theoretical artifact that advances and extends the existing knowledge of maturity assessments. It, therefore, has new facets that can be important to future research on the application cycles of maturity models.

From a practical and managerial standpoint, the article described a new method that can be adopted by organizations of all kinds and applied to any maturity model. The applicability and effectiveness of that method were validated by a case study, which proved that the artifact is ready for extensive practical use. In particular, the artifact will likely benefit organizations that use maturity models without proprietary assessment methods. The CMAM can be implemented as an organizational routine enabling up-to-date mapping of organizational capabilities. As shown by the study, this type of implementation makes it possible to identify improvements or deteriorations more rapidly, which means that plans for improvement processes and target setting may become more effective.

6.3 Limitations

Although it is rigorous, the presented work is not without limitations. Those limitations are related to DSR, which allows various artifacts to be developed in line with different preconditions and aims at the identification of suitable rather than optimal solutions (Hevner et al. 2004). First, the selection of existing assessment methods within the SLRs and the choice of experts for the formative evaluations influenced the design and the development of the CMAM. We performed structured SLRs in line with an established method to collect all articles that may have been relevant (vom Brocke et al. 2009). While selecting the expert panel, we identified researchers and practitioners from different domains who possessed advanced expertise in the design of maturity models. Another limitation arose from the generic design of the CMAM, which does not provide concrete references, tools, or questionnaires for assessing capabilities in specific fields. However, organizations can easily embed existing questionnaires into the CMAM or create quantitative data-acquisition methods internally. The CMAM is not intended to be too specific. Therefore, the CMAM need not replace proprietary assessment methods; they can be included in superordinate frameworks. Finally, the two summative evaluation episodes were limited to six organizations, and the CMAM was only implemented at one organization. However, despite this limitation, we deem the results of the evaluation to be generalizable because the CMAM is sufficiently generic to be applicable in different scenarios.

7 Conclusion

Given the wide variety and the high complexity of existing maturity models, users require comprehensive support during all phases of the application cycle. However, existing assessment methods do not provide sufficient support in all phases of that process and cannot be applied to all types of models. Furthermore, they do not consider activities or principles that are relevant to continuous assessment. This tendency runs contrary to the fundamental character of capabilities because organizations are embedded in ever-changing environments while their capabilities are liable to continuous change (Nelson and Winter 2004). Therefore, we propose the CMAM, which plugs existing research gaps and contains phases and activities that enable a generic, comprehensive, and continuous maturity assessment for all underlying models and all organizations. Drawing on the research gaps we identified, we formulated a fundamental RQ, which we translated into four concrete design objectives for the CMAM. We followed a rigorous research methodology, which is anchored in the DSR and enabled us to engage in goal-oriented analysis and synthesize indispensable principles and best practices from existing methods. Based on the formative evaluations, we iteratively extended these findings and created the CMAM. A summative evaluation of six market-leading organizations and an extensive six-month case study enabled us to determine that the artifact meets the design objectives. We are confident that the CMAM assists users during all phases of the application cycle and that it extends the descriptive knowledge of maturity assessments. As a general matter, the CMAM affects decision-makers in organizations, so the application of a maturity model becomes not a one-off or a point-in-time process but an embedded mechanism for continuous improvement.