1 Introduction

A wide variety of modeling methods have been introduced in order to capture, visualize, and analyze various aspects of an enterprise. Although enterprise modeling can provide clear benefits, every modeling approach has its advantages and disadvantages. It is beneficial for the enterprise modeling community to better understand the factors which make a particular modeling approach more or less appropriate in a particular context. Such experiences can be gathered through application of one or modeling methods to case studies.

In this work we report experiences applying three modeling approaches to industrial cases conducted in the context of Bachelor/Master thesis work. From these experiences in this context, we report benefit and drawbacks of individual methods in practice, as well as comparative experiences between some methods. More specifically, we report on modeling results for four cases (companies) using three modeling notations: two cases conducted using goal modeling (iStar) [5], one case conducted using goal and e3 value modeling [10], and one case conducted by a further group using goal and workflow modeling [7].

The case studies were conducted as part of a Chalmers Software Center project investigating strategic API design for software-intensive businesses. We focus on practical experiences with the three modeling frameworks, reporting qualitative experiences in order to address the following research questions:

Q1.:

What are the benefits and drawbacks of applying each modeling approach in our industrial context?

Q2.:

How do experiences with goal modeling and workflow modeling compare?

Q3.:

How do experiences with goal modeling and value modeling compare?

This work makes several contributions: (1) Our results allow potential modelers conducting projects in similar contexts to weigh the benefits and drawbacks of enterprise modeling approaches, facilitating informed selection. (2) Our findings outline improvements for the evaluated modeling languages, giving method developers feedback for future work.

This paper is structured as follows. Section 2 gives background on the modeling approaches and the cases. Section 3 outlines the methodology followed by the groups, while Sect. 4 contains the main observations of our studies. We discuss results and validity threats in Sect. 5, related work in Sect. 6, and provide conclusions and outline future work in Sect. 7.

2 Background

2.1 Modeling Methods

We give a brief introduction to the modeling techniques applied in our cases.

Goal Modeling. The cases used the iStar goal modeling framework, based on the recent iStar 2.0 Standard [5] consolidating versions of i* originating from [22]. Although we use iStar, we believe our observations would apply to qualitative versions of similar goal modeling languages (e.g., GRL or Tropos).

In iStar, goals are defined as objectives which the system should achieve through different tasks and dependencies on other actors [5]. Goal models include qualities, goals which are achieved or denied via positive or negative contribution links (e.g., helps, hurts). The model shows a hierarchy of goals assigned to actors, tracking the high-level goals of the system to low-level tasks, resources, or to dependencies on other actors. See Fig. 1 for example resulting goal models.

Workflow Modeling. Workflow refers to actions, a sequence of activities, or tasks set in a certain order enabling a system to serve its purpose [7]. The workflow focuses on the use of entities (e.g. actors and activities) to explain the sequence of operation or work procedures. In this study UML activity diagrams (with swimlanes for actors) are used to model the workflows of interest.

Value Modeling. The e3 value model has been developed to identify the exchange of value objects between different actors in a business network [10]. Value objects can represent a service, right or a quality attribute. Value models include actors, value objects which belong to at least one actor, value ports with value interfaces which are used by actors to exchange values, and value activities, which are the activities the actor must do in order to deliver the value object. In the case we used e3 models as per [10].

2.2 Research Context: Project and Company

The purpose of the ongoing project is to explore techniques for strategic API design, viewing APIs as a means to protect and expose business aspects. The Chalmers Software Center project works in six-month sprints, and has completed its second sprint focusing on API analysis and design [1]. This paper reports results from the most recent sprint, applying three enterprise modeling techniques to the elicitation and analysis of API designs for four companies. We leave reporting of API insights to future work, focusing on modeling observations. Although all companies were dealing with high-level API design challenges, each company was interested in particular analysis questions, leading us to choose differing modeling methods for each case (see API Challenges and Modeling Approaches in Table 1). All companies were interested in mapping their API ecosystem(s); given the existing work using goal models for ecosystem mapping (e.g., [20]), we applied goal models in each case. Company C3 was particularly interested in Workflow modeling, while C4 wanted to explore API value.

We summarize each company and their interests in Table 1, keeping identities anonymous. Each of the four companies are large (minimum 2000+ employees), with headquarters in Scandinavia and operations worldwide. The companies develop products and systems that include software, hardware, and mechanical components. For some companies, we modelled API systems which were in place or in development, in other cases the API framework was in the planning stages.

Table 1. Summary of cases for each company
Full size table

In terms of modeling background, company participants had a technical background and were generally familiar with modeling, including workflow modeling, but were not familiar with goal or value modeling. The students involved were senior students in a Software Engineering Bachelor or Masters (3rd or 5th years students) who had been introduced to UML modeling as part of their education, but also did not have explicit experience with goal or value modeling. G1 and G2 consisted of 3 Bachelor students, while G3 had two Masters students.

3 Methodology

We summarize the elicitation and modeling methodology of each of the three groups. More information (can be found in the full theses [4, 6, 18].

Elicitation Modeling. Each of the three research groups conducted a series of group and individual workshops and interviews in order to collect qualitative data to facilitate modeling. G1 and G2 conducted workshops with each company at the company location, as well as follow-up online interviews. The general elicitation process can be summarized as follows:

  1. 1.

    Introductory Group Interview: Necessary to understand the API ecosystem of the companies.

  2. 2.

    Off-site Modeling: Using available context knowledge and the scenario of focus (user story) to create initial model versions.

  3. 3.

    Interactive Workshop: Starting with the initial models, expand and correct the models interactively using group input.

  4. 4.

    Follow-up Online Interview(s): Finalize data collection and fill gaps discovered while modeling. Discussed experiences with modeling approaches.

  5. 5.

    Dissemination Workshop: Summarizing modeling and analysis results in a workshop with all company representatives present.

Workshops had 3–6 company participants including roles such as developer engineers, development managers, software engineers, product managers, and expert engineers. Online interviews were conducted with individual representatives from each company, who had been present in the workshops. Workshops lasted about three hours, while individual interviews were typically one hour.

G3 also conducted an introductory group interview and off-site modeling with C4, but gathered further data with individual semi-structured interviews and a survey, selecting participants involved in the API Framework (FW). Ten interviews and two surveys were conducted with the same questions, interviews lasting 45 min. G3 also had access to archival data concerning the FW.

Information gathered from workshops and interviews, including a selection of technical documentation, was used to create models. It was agreed with the companies that the first round of modeling should focus on a particular scenario or user story, to keep the scope of the models in check. Each group attempted to classify their resulting goal models in terms of a layered API architecture developed as part of the ongoing project (containing the Business Asset, API, SW App, and Domain Layers). The modeling process was iterative, with the students receiving iterative feedback from someone knowledgeable in the modeling approaches (the first author) and the company contacts.

G1 used goal model element colors as a means to indicate which elements were problematic, or which solved problems. In contrast, G2 had time to make explicit use of systematic qualitative goal model analysis to find issues in the current or planned design captured in the model [16]. In this analysis, each goal and softgoal in the model is given a label showing its satisfaction level, which is dependant on how the tasks in the model contribute to each goal.

Tooling. G1 and G3 used Microsoft Visio with appropriate stencils to make the iStar and Value models. G2 used Visual Paradigm to draw the workflow models. G2 drew iStar models in Creative Leaf [15], a free online iStar modelling tool, and OpenOME [17], an Eclipse-based open-source i* tool.

Model Validation. For G1 and G2 model creation was iterative and continuous, with workshops and interviews presenting and receiving feedback on the models. G3 explicitly used member checking to improve the accuracy, credibility and validity of the collected data [13]. Four new interviews were scheduled with previous participants, each lasting around an hour, during which G3 described and went through the models step by step, receiving feedback.

Feedback and general impressions on the process and modeling approaches were collected mainly by the students, via recorded interviews. Interviews were transcribed and feedback was coded to provide input to the theses.

4 Modeling in Practice: Observations and Comparisons

In this section we report the qualitative observations encountered by the three groups when using the modeling techniques in practice. As we used goal modeling in all four companies, but workflow and value modeling in only one company, we have more data to report on goal modeling. Figure 1 shows simplified goal models resulting from G1 for C2, showing the transition from the current to distant future and the use of colors to highlight issues (red) and solutions (blue). Here we can see the goal model divided into the API layers. The actual resulting goal models had 50+ elements, the workflow models had 10+ activities, while the value model had 20+ value exchanges. Further example models, including workflow and value models, can be found in the theses [4, 6, 18].

We report observations mainly from the viewpoint of the student groups conducting the modeling (G1, G2, and G3), but also observations and impressions from the company representatives, and occasionally the group supervisors.

Fig. 1.
figure 1

Simplified version of current and distant future for C2 (Color figure online)

Full size image

4.1 Goal Modeling

General Impressions. G2 observed that goal models captured the dependencies between actors in the API ecosystem and this provided valuable information to the company. G3 noted that while the goal models were initially perceived as difficult to read by the company representative, the details in these models got a positive response in later member checking validation. While the main purpose of the validation was to validate the data itself, the model was perceived as easy to understand quickly and displayed the relevant information clearly.

G1 noted advantages of capturing an API ecosystem in goal models rather than text, with a C2 representative noting “It is a big advantage that this is an image and not text... I see a lot of advantages with this”. A participant from C1 noted the benefits of having a high-level graphical overview. “benefits and strength (of goal modeling) is definitely giving the overview understanding... it is good that it combines all the elements so you can see the connection in the big picture”. The same participant confirmed the benefits of graphics over text “Once you get the goal modeling syntax, you understand all of this, otherwise you will have to use and write a lot of text to explain the same thing”.

One could argue that this observation – the benefits of graphics over text – is common to all models. Still, it is useful to confirm this result, particularly for goal modeling, showing that the model size and complexity does not cancel out the benefits of graphical visualization.

Recommendations. Goal models can initially be overwhelming, but with time and knowledge of the syntax, can provide a rich way to capture strategic data. Their selection should depend on the level and depth of modeling in a particular case: they are likely not appropriate for “light” or quick modeling, but more appropriate for cases where more extensive modeling can take place.

Validation. G1 and G2 went through a process of iterative validation, where models and associated questions (gaps) were sent via email to company representations, and feedback was received during online meetings. These groups noted difficulties in getting company time in order to validate the model, due to the time and resource constraints of the companies. G1 further noted that the syntax of the models had to be re-introduced with every discussion. Nevertheless, these groups did receive feedback which results in changes to the models; however, most of the feedback was not framed in model terms, leaving the groups to interpret them in terms of the model.

G3 had a more specific validation stage, selecting a subset of C4 representatives for model member checking, walking the participant through the model. In this case, participants generally agreed with the goal model. Participants were engaged enough to make a few corrections, adding a few goals, softgoals, and tasks, changing the responsibility for some tasks, and adding two new actors.

Recommendations. We observe that although model validation is generally possible, letting stakeholders look through the model on their own is likely not fruitful. Shared experiences show that it is more useful to walk through the model with stakeholders. Of course this is time consuming, G3’s strategy of selecting a sample sub-set of stakeholders for dedicated verification sessions appeared successful, and can be recommended when context allows.

Qualitative Models. G2 in particular expressed frustration with the qualitative nature of goal models, particularly with conflicting contribution links between (e.g., a help and a hurt to the same softgoal). This was also the case when using systematic forward evaluation, where human judgment was used frequently to resolve evidence. For example, in a case where several “hurt” contribution links make a softgoal denied, or partially denied? G2 felt these decisions were made subjectively, and often used a conflict label.

Recommendations. G2 felt quantitative analysis could be more helpful, and G1 made a similar recommendation. Goal modeling should support quantitative data and analysis whenever possible.

Layers Views. The groups reported experiences dividing the goal model into API layers. Overall, G1 and G2 found that using the layered API framework with the goal model increased the readability of the models. G2 noted that structuring the actors into clearly defined layers made them easier to locate, both while communicating about the models and during the modeling. G1 agreed, the C2 contact also provided positive feedback on the layers, with a C1 representative adding “It is much more useful to make it layered... otherwise it just gets, it’s just a big spiderweb that you look at and think that it is too much.”.

However, G2 noted that layers limit the placement of dependency- and contribution links, particularly when there are many links between the top and bottom layers. G1 noted a further limitation in that the reader must understand the purpose of each layer, requiring additional training or explanation.

G2 found that it was not always obvious where in the layered API framework a specific actor belongs, this often depended on perspective. In fact, G1 created different layer configurations for different perspectives. G1 had some initial negative experiences mapping actors to layers in the initial stages, when their understanding of the domains was incomplete.

In addition to using layers, G1 dealt with the complexity of their models using views, sub-sets of the models captured in separate files, receiving positive feedback from the company contacts. However, G1 noted that the larger context of the entire API ecosystem must be considered – the views cannot replace this model. The C1 representative noted “I think that whatever comes up in those discussions needs to go in there. Because some of them will prove that you have some connections that might show up to be problematic.”.

Recommendations. Layers and views are helpful for managing and highlighting key points on large and complex goal models. However, each layer must have a clear semantic meaning, communicated to participants; layers should be added in later stages, when domain knowledge is more complete; and, modelers should be aware that placement of an actor in a layer may change depending on perspective. Views cannot replace the full complexity of the model.

Complexity and Completeness. When it comes to the complexity of the goal models, C1 and C2 provided similar feedback, stating that it is both a strength and a weakness. One participant stated “... it becomes messy and complex and I don’t know in terms of complexity what the upper limit is”. Participants also noted that knowledge was cumulative, based on past sessions, “... for those who haven’t been in discussion of the layers, it would be hard to discuss the business assets and APP SW as separate layers”.

G1 noted that modeling with the companies helped trigger discussions revealing information that was not available in the documents received. However, G1 also noted that it was difficult to determine when the model should be considered complete. As the companies cooperate in the modeling process, new gaps were identified and the modeling continued iteratively. A C3 participant stated “There is all the information here, but it is a little bit messy”.

As they iterated over the models, G2 found that they had to make decisions based on a trade-off between completeness and simplicity. The models should be complete enough to be useful for analysis. As the group added more actors and elements to the model, the models became more difficult to understand, which had a negative impact on the effectiveness of the model.

The groups noted the presence of a goal model learning curve, a C2 participant explained “it is hard to understand for people who don’t know goal modeling”. G1 noted that they had to continuously share a document reminding the stakeholders of the syntax of goal modeling and explanation of the different layers used. The participants for G2’s case made the point that differing levels of understanding were needed depending on the level of involvement in modeling. However, a C3 participant stated “I wouldn’t say I feel that is any harder than some other modeling languages.”.

Recommendations. Some complexity is unavoidable. One recommendation is to create an initial large model which covers much of the domain, then remove actors not involved in the analysis. The learning curve can be mitigated with training, and the level of training can vary depending on participation.

Scope. G2 noted that they limited the scope of the goal models by excluding actors which are judged to be unrelated to the changes implemented in the to-be model. Both G1 and G2 focused their early modeling efforts on a particular use case or scenario provided by the companies, while G3 focused only on modeling the existing FW. This early material was necessary to help the groups focus and start modeling, but the later modeling diverged from this initial scope. In the case of C1, the scenario description was quite brief, and the resulting model was relatively small, resulting in a significant expansion of the model in the interactive workshop. In the case of C2, the initial user stories were long and detailed, resulting in relatively large starting models, much of which was removed or changed as a result of in later scoping discussions.

Recommendations. Setting an initial scope for modeling is essential, starting the scoped modeling process from an existing use case or scenario, ideally documented, is also a good practice. However, it may be better to limit the size of this starting point, to avoid modeling effort which is deemed out of scope, and to make it easier for participants to understand and contribute to the model.

Methodology. G1 commenting on the lack of explicit methodology for iStar. Although some methods exist (e.g., [16]), there is no one established method purported by the iStar 2.0 standard. G1 recommends the method in Sect. 3 as a starting point for those working with goal models in an industrial project.

Recommendations. Experiences in this study indicate that a general goal modeling method is needed.

Highlighting and Analysis. G1 used element coloring to show problematic and “fixed” model elements. This helped to show the improvements and transitions through planned API deployment in the C2 case. In the C1 case, where time transition was not emphasized, the colors were less helpful. The goal modeling expert/supervisor noted that the colors did not always reflect the semantics of the model. For example, an element would be marked by the group as red, when the incoming contribution links were all positive (help). G2 used systematic goal model analysis (i.e., [16] with some success. They were able to use the analysis to find unfulfilled softgoals, which helped them to ask relevant questions and enabled them start thinking about ways to resolve unfulfilled elements.

Recommendations. We avoided introducing systematic evaluation for G1 mainly to avoid overwhelming them with detail (they were covering an additional case compared to the other groups). In retrospect, it seems this analysis was generally usable by the G2, and in fact G1 came up with their own, less systematic and less semantically consistent method to make similar points using the models. Given these experiences, we can recommend the user of systematic analysis for student modelers conducting case studies.

Interactive Modeling. G1 conducted explicit interactive modeling sessions with two companies (G3 did offline modeling, while G2 used their second workshop to iterate over their current models with one company representative). G1 noted that every participant adds their own insight and perspective, and the goal model can quickly drift away from the original scope. This was apparent during the C1 workshop where questions like “is this relevant to the API ecosystem?” or “is this considered inside the scope of the API ecosystem?” were frequently asked. However, company participant expressed support of the interactive workshop “... it is good if there is some kind of workshop and brainstorm and everyone knows the syntax and then build them together.”.

Recommendations. Interactive workshops build model content, stakeholder investment and familiarity with the model. However, participants should be given initial training, and care should be taken to direct and focus the discussion.

Tooling. When selecting a goal modeling tool, G1 and G3 used Microsoft Visio, in part because it is a stable commercial tool, but also because it allows flexibility in terms of syntax and notation, e.g., element colors and splitting the model into layers. G2 had trouble acquiring an academic license for Visio. They started with the Create Leaf online tool, but eventually moved to the desktop Eclipse-based OpenOME tool. They did so because they felt their increasingly larger models were more visually pleasing in OpenOME.

Recommendations. iStar-specific tools should allow more flexibility in terms of what can be drawn. Care should be taken for the visual appearance of models.

4.2 Workflow Modeling

General Impressions. G2 felt that the workflow models were important tools for the researchers to understand and get an overview of the current and planned workflow. The models effectively communicated how different actors in the API ecosystem interact. However, they were not frequently used in the analysis of the ecosystem once the workflow was understood by the researchers. The aspects captured by the models did not provide new insight for the case company, i.e., they only modelled what was already known.

Recommendations. Workflow models are useful in early project stages to understand actor interaction, particularly for analysts to confirm their domain understanding.

Layering. G2 attempted to apply the API layers to the workflow models as well as the goal models, but had difficulties. They considered using swimlane notation for layers, but this would conflict with their use of swimlanes for actors.

Recommendations. Instead of graphical layers, one could try other visual indicators, like colors, for adding layers to workflow models. Else it seems the structure of workflows are less amenable to classification and layering of elements.

Analysis. G2 did not perform any sort of systematic analysis on their workflow models, e.g., model checking [7]. This was mainly due to: (1) the learning curve and tool support barriers for bachelor students to understand and apply this type of analysis, and (2) difficulty to get the specific historical or constraint information needed for such analysis from C3.

G2 did note a desire to capture action duration (the time it takes to complete an action) in their workflow models, in order to support an analysis of potential bottlenecks. They believe that the ability to tell at a glance where the time is spent would assist in the development of an improved workflow. Generally, approaches to address duration or time in workflow BPMN models do exist (e.g., [9]), but again, these approaches are difficult to apply without great effort in the context of an industrial bachelor thesis.

Recommendations. Formal or quantitative workflow analysis should be more accessible, particularly via tools. An easy way to capture task duration is needed.

Workflow vs. Goal Modeling Generally, the use of workflow and goal models together provided analysis value, representing two different ways to look at the same thing (the ecosystem in terms of goal fulfillment and the ecosystem in terms of activities). The use of workflow models in addition to the goal models made it possible to gain an initial understanding of the underlying process of profile (/API) development of the company.

As mentioned in Sect. 3, G2 was able to exploit the links between workflow and goal models. Although this link can be exploited in either direction G3 chose to start with workflow models and use them to help create goal models, inspired by existing work [21]. Specifically, tasks in the goal models correspond to the actions in the process models, and the actors in the goal models are represented as swimlanes in the process models. They also used these links to help maintain consistency between the models. G2 notes that in order to exploit these links, the vocabulary between the models must remain consistent.

G2 notes that more focus was put into the workflow models at the start of the elicitation. These models worked largely as an initial one-way communication tool between the company representative and the researchers: G3 verified their understanding of the workflow by asking for feedback. During the latter half of the study, most of the attention of the group of researchers and the company went to the goal models. The natural tendency within the study was to use the two types of models sequentially rather than together. According to G2, once the goal models had been finalized, the process models did not offer much valuable information for the analysis of the company’s API ecosystem.

Although iStar is not designed to be temporal, G2 noted that it was easier to represent time constraints in goal models using softgoals such as “fast documentation” and “fast approval”. Furthermore, G2 noted that goal models were able to imply a temporal ordering, even though this is not explicitly part of the language. For example, the necessary sequence of actions can be implied with dependencies, if A depends on B, B must be done before A.

Recommendations. Tooling should exploit the links between the two types of models, helping modelers to keep the vocabulary and models consistent, and partially auto-generating one model from the other. Our experiences showed that workflow models were more useful at the beginning of the elicitation and analysis process, as the researchers were trying to get a handle on basic domain flows, while goal modeling was more useful later in the process, when domain understanding had increased. Inferring temporal information from goal models as is may be problematic, as this information may be interpreted differently by different modelers. However, if this implicit ordering can be clarified, standardized, and made consistent with workflow models, particularly with tool support, this can further enhance the use of goal and workflow models together.

4.3 Value Modeling

General Impressions. G3 did not explicitly ask for feedback about the models from C4 company representatives; nevertheless, while member checking the model, they received some feedback. Generally, the feedback during the member checking process was positive. While the main purpose of the validation was to validate the data itself, the models were perceived as easy to understand quickly and displayed the relevant information clearly.

Value vs. Goal Modeling. G3 reports that their industrial supervisors initially liked the value model more than the goal model. However, this model was presented in a shorter status update meetings, where they did not go through the goal model in a step-by-step manner. As the value model is more straight forward and smaller, G3 found that they needed less time presenting it to make someone grasp the model, compared to a goal model.

G3 used the goal model as a means to compliment the value model: the goal model visualized the identified challenges and values and maps them to the relevant actors. Tasks or lack of tasks can show why these challenges and values exist. G3 made an explicit link between the actors in the goal and value models, but did not explicitly link values to any iStar elements. Overall, G3 found that applying e3 value modeling did not find new information compared to analysis of the goal model, it only presented the values gained in a more direct and abstract manner. The value model can serve as a quick visualization of which stakeholders gain what value, but it does not explain why or how they gain value.

Recommendations. Value and goal modeling can be seen to be complementary, with value models showing a simpler, more understandable view of actors and value exchanges. However, goal models were more effective at assigning values to actors, and to capturing the “why” for values, although this comes with a complexity price. As with workflow models, tooling could better support the link between goal and value models, linking actors to actors, and mapping values to one or more iStar elements.

5 Discussion

We can summarize our results to answer Q1–3. In terms of benefits and drawbacks of industrial application (Q1) goal models were found difficult to understand at the beginning of a project, with a significant learning curve. However, the longer-term results and impressions were more positive. Goal model validation is possible, but time consuming. The models are complicated and capture much information, this is both a drawback, and a strength, as capturing the same magnitude of information with text would be daunting. The groups saw the qualitative nature of goal models as a drawback, even though it is unclear how easy it would have been to gather quantitative data in the cases. The groups complained about the lack of explicit method for goal modeling. Our results show that systematic evaluation for goal models is an advantage, but that goal model tooling can still be improved. Interactive group modeling is possible, but only with attention to focus and participant training.

Workflow models were helpful at the start of the project to clarify actors and flows, but the group found they were missing specific duration information, and could not easily apply any analysis to find or show bottlenecks. Value modeling was initially quite easy to understand, and helpful as a summary view.

Comparing goal to workflow modeling (Q2), our evidence supports the idea that these models are complementary. Workflow models clearly show process, which is only informally implied in the goal models. However, workflow models were not as helpful in analysis, and proved more useful later in the project. Similar results were found for value models (Q3), they were initially helpful as they were easily understood, and were seen as a helpful summary view as compared to the goal models; however, the goal models were seen as more helpful for analysis, with the value models not providing any new insights.

Our experiences lead to a number recommendations. Goal modeling is useful for projects with sufficient modeler and stakeholder time, for a deeper analysis of motivations. For shorter projects where motivation or dependencies are understood, this type of modeling is less applicable, while value and workflow models could be appropriate. Workflow analysis would be improved by the ability to easily capture task duration and by more accessible analysis techniques. When there is sufficient time or need, the combination of these types of models is beneficial in practice. Available tools, which are accessible to students and case-study-ready, should better support links between these types of models, supporting consistency checks and partial derivations.

Threats to Validity. In terms of Construct Validity, it is possible that the student modelers and the company representatives misunderstood the syntax or semantics of goal, workflow, or value models. We mitigated this threat by giving an overview of the modeling language or the models themselves at multiple points in the study. In particular, we note that G3 did not use the calculations available with e3 value modeling (e.g., [10]); however, collecting cost measures for an internal API framework would have been challenging.

Considering Internal Validity, all groups used some form of triangulation, collecting data from workshops, documents, archival data, and interviews. For C1, C2 and C3, validation rounds after the interactive workshop only involved one person per company. However, we mitigate this effect by collecting impressions from more than four people from four different companies in different areas.

Examining External Validity, all case companies are located in Scandinavia; however, these companies are large and international. Applying the same modeling process to different companies with different challenges, different workshop participants, and different validation stakeholders may produce differing observations. However, we mitigate this possibility by involving a number of people from four different companies. The focus on strategic API modeling may bias results; however, we feel it was useful to focus on a concrete challenge and problem within each case. In a different case with more complicated or varied workflows, workflow modeling may have been perceived as more useful.

Finally, considering Reliability, our study had the participation of a goal model expert, a co-author of the recent iStar 2.0 guide. For demonstration purposes she made one of the first versions of the model for one of the user stories provided by C2. The other user story and all other models in the studies were created by the students. The students received feedback on goal model syntax and semantic issues, but most changes were on the small-scale, e.g., wrong link direction, wrong element choice, etc. The majority of the detailed impressions of the models and modeling experience were elicited by the students from the company directly, with little participation of the supervisors. The students and company representatives report both benefits and drawbacks of goal modeling, and most of the data reported in this work is from their perspective.

6 Related Work

Goal Models. Existing work has reported experiences with goal models in practice, for example the iStar Showcase [2] contains summaries of a number of case studies using goal modeling, a number of which mention scalability, complexity, and learning curve as iStar issues. Our experiences confirm these findings; however, we are able to give further recommendations for improvements based on our experiences, without significantly extending and complicating the language.

Earlier work from Estrada et al. [8], applying iStar in industry, find that it is expressive and applicable to the domain, but does not well support modularity, scalability, or complexity management. We find the same issues, but work towards modularity solutions with our practical experiences in layers and views.

Goal and Workflow Models. Work exists evaluating workflow models, although it seems much of this work focuses on controlled experiments (e.g., [19]), or on evaluating a specific aspect (e.g., configurability [12]).

Much work has looked at goal models and some form of workflow model (process, BPMN) together, typically with a focus on methods which map or transform elements from one model to the other (see [14] for a recent survey on goal model mappings/transformations). Few papers report practical experiences with basic workflow models, [14] reports that 92% of the papers found were (technical) solution papers, with evaluation focusing on new solutions.

G2 were particularly inspired by de la Vara et al. [21], who provide guidelines on deriving goal models from process models. This derivation method has been used to achieve consistency between the goal and workflow models in our study.

Goal and Value Models. Horkoff et al. find six papers using goal models with value models [14]. Most of these papers introduce rules, mapping, and methods to explicitly link goal and value models (e.g., [3]). G3 did not explicitly use any of these approaches, mainly as an evaluation of these methods was not the focus of their thesis. Work by Gordijn et al. links goals and e3 value model, making deeper use of their connections for iterative modeling than G3 did in their case [11]. This is due in part on G3’s focus, on analyzing benefits and drawbacks of the C4 FW, and not explicitly on evaluating the models.

7 Conclusions and Future Work

We have presented experiences applying three different modeling languages to analyze strategic API development for four case companies. Our experiences can help enterprise modelers to choose between the modeling languages, depending on their analysis needs, expertise, the depth of the required study, and the time available. We also provide feedback for improving the language, methods, and tools associated with these approaches, with an emphasis on goal modeling. Future work should implement and evaluate our recommendations.

Overall, feedback from the case companies given at the final cross-company workshop was positive, with the modeling providing sufficient insights on strategic API ecosystem development. Although positive feedback was collected for the goal and workflow models, the companies are particularly interested in ensuring API value and having high-level summary models. Thus the next sprint will focus on modeling and measuring value.