Keywords

1 Introduction

Enterprise Architecture (EA) is an instrument for decision makers to structure and manage organizations from an integral perspective. EA provides a holistic view of the organization, including customer offerings, business processes, information systems, technical infrastructure and the relations between these aspects. The purpose of EA is twofold: on the one hand it provides insight into the actual state of the organization, enabling the organization to determine the impact of changes. On the other hand it gives direction to such changes by sketching the design principles and designs that best fit the organization’s ambitions and goals. EA is the bridge between strategy and execution [1].

EA is seen as an instrument for organizations to achieve their business goals. The argument behind this view is that a well-structured, well-aligned organization is more cost-effective, agile and effective. The actual benefits of EA have been subject to academic research by different authors. The number of benefits claimed by authors is large, though proof of actual benefits is less abundant [2, 3]. For example, Boucharas et al. [2] found in a structured literature review 107 academic publications mentioning benefits, of which 33 were found relevant to the question of relating EA to benefits but only 14 fulfilled the qualitative requirements of the literature review. In these 14 publications, 100 different benefits are mentioned. In recent years various literature studies [2–7] as well as empirical studies on actually achieved benefits have appeared [8–10]. In these publications all authors define EA benefits in their own way. Where most authors introduce some kind of categorizing of benefits, these categorizations differ between authors as well. This lack of a common framework of EA benefits makes it difficult to compare different studies and is an obstacle in augmenting other research results.

In an earlier paper [11] we introduced a generic framework for classifying EA benefits, the Enterprise Architecture Value Framework (EAVF). We have used this framework as a starting point for a survey concerning perceived benefits in organizations. The first results of this survey [10] not only provide an interesting insight into the kind of benefits that are actually perceived within organizations, but they can be used to develop the EAVF into a EA benefits measurement instrument as well.

The research question we aim to answer in this paper is: Is it possible to develop an EA benefits measurement instrument based on the EA Value Framework?

In the next section of this paper we sketch the theoretical background to our research question, followed in Sect. 3 by an overview of the research method used in further developing the EAVF and the derived benefits measurement instrument. The results are presented and discussed in Sects. 4 and 5 and followed by conclusions, limitations and further research in Sect. 6.

2 Theoretical Background

In the literature, no common framework for classifying EA benefits can be found. The framework we developed in our research [11] is based on two theses:

  1. 1.

    Organizations benefit from EA when EA contributes towards their business goals.

  2. 2.

    Benefits may evolve from the inception of the architecture towards the implementation of architectural designs.

For the contribution towards business goals, we decided to use the four well-known categories of the Balanced Scorecard [12, 13]: the Financial, Customer, Internal and Learning and Growth perspectives as many organizations use these to classify their goals and it has been used by other authors to classify benefits as well [2, 5].

In order to follow the evolution of benefits in time, we introduce the lifecycle of EA in which we distinguish three main phases:

  • the Development of the architecture where principles and models are developed and registered. In this phase, usually the architects are leading;

  • the Realization phase where architectural designs are implemented and projects have to comply with the architecture. In most enterprises, project managers are in the lead in this phase;

  • the Use phase, where (parts of) the new architecture have been implemented and used in operations. In this phase, the actual operational benefits are obtained and the lead is with business line managers.

The idea of benefits developing in time can be found with other authors as well. For example, Foorthuis et al. [8] explicitly distinguish benefits in the project execution phase from other benefits where Tamm et al. [3] distinguish between benefits flowing directly from EA and benefits resulting from the implementation of EA plans.

Combining the two mutually independent axes results in the EA Value Framework (EAVF) as depicted in Fig. 1. The EAVF essentially divides the field of EA benefits in twelve classes of EA benefits: four perspectives times three phases.

Fig. 1.
figure 1

The Enterprise Architecture Value Framework

Full size image

The EA benefits measurement instrument we are developing is based on this framework and essentially consists of a series of questions that may be used to determine the perceived and realized benefits in every cell of the framework. These questions are derived from benefits as reported in the literature, especially from the work of Boucharas et al. [2]. Examples of these questions are given in Plessius et al. [11].

3 Research Method

In order to validate the EAVF and the EA benefits measurement instrument we conducted a survey. In this survey we defined for each cell in the EAVF one overall statement representing the class of benefits corresponding with that cell, as well as a number of questions representing the specific benefits belonging to that class. For instance, for the Learning and Growth perspective in the Realization phase we defined the main (class-representing) statement as:

  • By applying Enterprise Architecture in projects the learning and innovative capacity of the organization is better.

with the following questions on specific benefits in the class:

  • Projects carried out under architecture provide a better understanding of the limitations of the solution.

  • Projects carried out under architecture feature a more substantive decision-making process.

  • Projects carried out under architecture feature better sharing of knowledge.

  • Projects carried out under architecture more often produce results that fit the operational management.

  • Projects carried out under architecture produce more agility (flexibility).

The questions that ask about specific benefits, can be regarded as reflective measures of each main statement, giving a generic view on that class of benefits. In Sect. 5.2 we will examine which questions are most representative for each class.

All statements and questions were scored on a 5-point Likert scale. The survey was targeted at stakeholders of architecture in organizations. We included a question to be able to discern between the three roles that correspond with the rows of the EAVF:

  • Developers of architecture such as enterprise and domain architects.

  • Implementers of architecture, such as solution architects, designers, developers and project managers.

  • Users of architecture such as business line managers, IT managers and staff.

Based on their answer to this question, the respondents were presented with the questions on benefits related to the corresponding row. We included some questions on the background of the respondents as well. The survey consisted of 97 questions and in this way less than 50 questions were presented to all respondents.

From over 3000 mailings we received 287 fully completed responses where 110 respondents have answered the questions on the Development of architecture, 68 on the Realization of architecture and 109 on the architecture in Use. Based on the general questions on their background, we found the characteristics of the respondents congruent with the results found in other surveys (see for example [8, 14]) and as they are encountered in practice.

For the statistics in the next sections we have used SPSS edition 22 (Statistical Package for the Social Sciences, nowadays an IBM product). In most questions, the extremes of the Likert scale were hardly used and in order to reduce the number of possibilities - especially for the regression analysis (as described in Sect. 5.2) - we decided to bundle the answers in three categories:

  • (very) negative benefits reported (Likert categories 1 and 2);

  • neutral, neither positive nor negative benefits reported (Likert category 3);

  • (very) positive benefits reported (Likert categories 4 and 5).

By combining the original answers in these three categories we reduced the original questions to questions if benefits could be reported and if these benefits were deemed positive, negative or neutral. Given the small number of extremes in the original answers, we consider this reduction justified.

The survey has been carried out in the Netherlands with statements and questions in Dutch. For this paper all statements and questions have been translated into English, but there may be slight differences in meaning between the translated statement or question and the original one.

4 Benefits Perceived

In this section we present the results of the survey providing an overall picture of the kinds of EA benefits (positive and negative) that are actually perceived by organizations. Next, in Sect. 5, we will use the survey results to validate the EAVF and the survey questions as a measurement instrument for the twelve EA benefit classes.

4.1 Statements on the Benefits Classes

All respondents – regardless of their role - answered the twelve generic statements for the twelve classes of the EAVF. The results are presented in Fig. 2 where the numbers in each cell are the percentage of respondents who found that EA had a positive effect in that particular area, respectively found no effect of EA or found a negative effect of EA. The numbers are statistically significant as shown in Plessius et al. [10].

Fig. 2.
figure 2

Perceived benefits of Enterprise Architecture in the EAVF

Full size image

From Fig. 2 it is clear that – except in the Customer perspective – respondents perceive an overall positive effect of EA. Even where the percentage of positive responses is less than fifty percent, the overall effect is neutral rather than negative. These effects are consistent over the three roles: it seems there is consensus between stakeholders on the benefits of EA in each cell of the framework.

4.2 Questions on Specific Benefits

In all, we asked 70 questions about the occurrence of specific benefits distributed over the twelve cells. Of these questions, 17 questions did not show a significant result in the one-sided binominal test we performed (p < 0.05). These questions are not included in this section. In the following tables we present for each row in the EAVF the 3 questions that received the highest percentage of (very) positive answers and the 3 questions that received the lowest percentage of (very) positive answers.

In the development phase (Table 1) we find that the benefits perceived by most respondents are concerned with providing insight. The benefits perceived the least are related to the effect of EA on governance. One might conclude that the development phase provides insight, but that to turn these insights into decision-making lags behind. This is in line with previous research [8]. Still, more than fifty percent of the respondents indicate perceived benefits for each of the bottom 3 benefits.

Table 1. Top and bottom benefits perceived in the development phase
Full size table

In the realization phase (Table 2) we find a similar distinction. The top 3 contains benefits concerned with insight, while the bottom 3 consists of benefits related to actual project performance. Architecture does seem to contribute to better decision-making at the project portfolio level, but at the level of cost and time of individual projects EA does not seem to generate improvements.

Table 2. Top and bottom benefits perceived in the realization phase
Full size table

In the use phase (Table 3) we find a less clear-cut situation. The alignment between business processes and IT is in the top 3, but better cooperation within the organization is in the bottom 3. It seems as though business and IT have started to communicate with each other, but there is still space for improvement. Clearly the respondents see no effect from EA on market shares for most organizations.

Table 3. Top and bottom benefits perceived in the use phase
Full size table

The results shown in Tables 1, 2 and 3 are in line with previous research [8]. As the responses seem representative for the field, they present a good starting point for validating the EAVF and the measurement instrument based on the EAVF. We will discuss this in the next section.

5 Evaluation

5.1 Propagation of Benefits

From the meaning of the EAVF dimensions it may be expected that there exist positive relationships:

  • Horizontally from right to left as the Balanced Score Card argues that results in the learning and growth perspective should impact the customer and internal process perspectives, whereas the latter two should impact the financial perspective.

  • Vertically from top to bottom as the Architecture life cycle implies that results from the development phase should impact the realization phase and the results from the realization phase should impact results in the use phase.

We tested if these relationships hold in the EAVF as well by calculating the correlations between cells horizontally and vertically. Figures 3 and 4 show the Spearman’s rho values found with p < 0.05.

Fig. 3.
figure 3

Horizontal correlations between cells in the use phase

Full size image
Fig. 4.
figure 4

Vertical correlations between phases

Full size image

The horizontal relationships (between the perspectives of the Balanced Scorecard) were only tested for the use phase, as this is the phase in which the end results of EA are realized. The results in Fig. 3 show a moderate correlation from the customer and internal perspectives with the financial perspective. The relation between customer and financial is largest. This stresses the importance of the customer perspective, which in practice often gets little exposure (see Fig. 2).

Looking at correlations between the phases we find correlations with a Pearson’s rho > 0.300 between most phases. The correlation between development and realization in the internal perspective is lowest.

The correlation results seem to support the underlying assumptions of the EAVF (EA benefits can be related to organizational goals and the benefits may evolve in time).

5.2 The EA Benefits Measurement Instrument

In order to develop the benefits measurement instrument we researched if the questions defined for each class cover the main statement of that class, or, stated differently, can we predict the outcome of the main statement (the view on the EAVF-class as a whole) from the corresponding questions (the actual benefits in that class)? If this is possible, the questions form a sound basis for a questionnaire.

To research this question we used the method of ordinal regression for each cell with the main statement as dependent variable and the questions as independent variables. The link variable used is the logit as the distributions were varying across different cells and we wanted to use the same link function for every cell.

For each cell we built several models; starting with individual questions we took the best fitting question and added questions while the prerequisites were satisfied. The prerequisites we used to accept a question in the model are: significance of model fit < 0.05, Pearson’s goodness of fit > 0.05 and significance of parallel lines > 0.05.

In ten out of the twelve cells of the EAVF we found a relation between the dependent variable (the overall statement) and some of the independent variables (the questions). As a threshold for acceptance we used a Nagelkerke pseudo R2 > 0.250, which in itself is low but can be defended as this is a first try at validating the instrument and we did not want to reject possible relations prematurely. For discussion purposes we present here one of the results (Table 4), corresponding with the Learning and Growth perspective in the Realization phase. All results can be requested from the authors.

Table 4. Model example
Full size table

As can be seen from Table 4, from four out of the five questions around 50 % of the overall statement can be explained, whereas for the fifth question no statistical evidence was found, as adding this question gave rise to a quasi-complete separation of data.

In most cells, adding the excluded questions to the model made the model fit statistically not significant (p >= 0.05) or the test of parallel lines failed. Adding more responses could help to overcome this. In Fig. 5 we have summarized the results in the framework, where the number gives the Nagelkerke pseudo R2 of the best-fitting model found.

Fig. 5.
figure 5

Model fit in the cells of the EAVF

Full size image

In the empty cells there seemed to be a quasi-complete data separation for all individual questions, so we could not use ordinal regression for model building.

6 Conclusions and Further Research

The contribution of this paper is twofold: we introduce a framework, the EA Value Framework, for classifying EA benefits that combines the aspects of goal and time. To be able to compare results from different EA benefit research initiatives and to enlarge our knowledge base on EA benefits by building on each other’s research, it is important to share a common framework. Supplementing this framework we show the current state regarding EA benefits in the Netherlands, based on a survey held in the first months of 2014.

The results of this survey show the kind of benefits organizations experience at the moment. The main conclusions we can draw from the results are first of all that regarding the customer perspective, benefits reported are low. This is consistent with findings in the literature (for example, Boucharas et al. [2] found only two benefits in the Customer perspective out of 100 benefits) and our observation that many architects are focused on the internals of the organization (processes and information) and not on the relation with the outside world. Secondly, we found that most benefits seem to occur in the Development phase, which may be caused by the fact that in the Realization phase project managers may perceive EA primarily as a constraint instead of a support and in the Use phase results can not be attributed to EA only. In future research we want to explore these hypotheses in case studies.

The results of the survey appear to be representative of the EA field (as discussed in Sect. 3), so we used them to evaluate the EAVF as well. We found moderate support for the assumed underlying relations between the cells of the framework which in turn give support to the validity of our framework.

Finally, we used the results to continue the development of an EA benefits measurement instrument based on the EAVF. This instrument consists of a series of questions for every cell in the EAVF, which are derived from benefits as reported in the literature. These questions have been used to predict the overall outcome in the cell, as measured in an overall statement for that cell. Using ordinal regression, we found valid models for ten of the twelve cells. These models constitute the first step in defining a final questionnaire to measure actual benefits for a specific cell.

Our research has its limitations. As our survey asks for the perception of the respondents concerning EA benefits, the outcome is subjective. This is a frequently occurring phenomenon with evaluative surveys but there are indications that this kind of survey leads to reliable results. For example Wall et al. [15] show that perceptions are a reliable indicator of actual organizational performance. Secondly, the respondents to our survey are self-selected and therefor are not a random sample of the EA community. As a consequence some bias in the answers may be present. Moreover, as the survey is conducted in the Netherlands, care must be taken in generalizing the results. Finally, as the twelve main statements are generic by nature, they leave room for different interpretations.

In order to examine if the overall statements cover the cells fully and to further refine the results from our survey, we plan to perform case studies in organizations with the EA benefits measurement instrument. In that way, we expect to get a better understanding which benefits are most important for organizations and gather ‘best practi-ces’ on how to maximize the benefits of EA.