Keywords

1 Introduction

Delivering business value by agile software development (ASD) teams is one of the core concepts within the agile domain [1]. The first principle of the agile manifesto is: ‘Our highest priority is to satisfy the customer through early and continuous delivery of valuable software’ [2]. However: ‘Satisfying the customer through continuous delivery of valuable software is not an easy task’ [3], says Jim Highsmith; one of the manifesto’s authors.

Scrum is one of the most commonly used agile frameworks [1, 4, 5]. The fundamental unit of Scrum to deliver business value is a small-sized ASD team with a distinct division of roles. It consists of a scrum master, a product owner, and several developers [5]. The same is true for the agile teams mentioned in the most used agile scaling framework, SAFe [6]. Both frameworks prescribe that the product owner is accountable for maximizing the value delivered by the ASD team. Ultimately, in ASD the product owner is accountable for business value delivery to the stakeholders.

To maximize business value delivery, Scrum and SAFe contain a value delivery process [5, 6]. This process starts with putting requested value items on the product backlog. In most cases more details are required for the product owner to be able to prioritize these items. The ASD team also require details to make sure they deliver the requested value. These details are collected and added during the refinement by the product owner, in close collaboration with the stakeholders, and the ASD team. Based on the outcome of the refinement, the product owner can (re)prioritize the product backlog. The ASD team will take the highest item(s) from the product backlog to be delivered. Before, during, and after delivery, validation will take place with the stakeholders to determine if the requested value is delivered. Scrum and SAFe define the validation as a sprint and iteration review. A review is an informal acceptance of the solution potentially supported by measuring the value delivery with metrics [7].

The contribution of this research is to explore how product owners operationalize business value. Our main research question is: ‘How do product owners operationalize business value delivery with their agile software development teams?’ Based on the process of value delivery, four sub-questions are identified:

  • RQ1: How do product owners determine the most valuable backlog items?

  • RQ2: How do product owners refine business value?

  • RQ3: How do product owners validate business value delivery?

  • RQ4: How do product owners measure business value delivery?

This paper is structured as follows: In Sect. 2, we present related work. Section 3 elaborates on our research method. The results of our research are presented in Sect. 4. In Sect. 5 we report our validity threats. Section 6 discusses our main findings and suggests future research opportunities. Section 7 summarizes our conclusions.

2 Related Work

In this section we present related literature on the four activities carried out by the product owner in the value delivery process [5, 6]: 1) prioritization, 2) refinement, 3) validation, and 4) measurement.

Prioritization

Because there are usually more requirements than feasible, given budget and schedule constraints, prioritization by the product owner is important to select the most valuable product backlog items [8, 9]. Scrum does not prescribe a prioritization method [5], therefore different methods are used. Hujainah et al. [9] identified 108 prioritization techniques. Jarzębowicz et al. [10] found that among the 69 Polish IT practitioners investigated, 4 prioritization methods are used. SAFe recommends a method called Cost of Delay (WSJF) [6]. Rodríguez et al. defined 47 value propositions used in value-based feature selection by interviewing 21 key stakeholders from 3 companies [11]. Other research shows that prioritization is only partly understood [8, 12, 13].

Refinement

A product backlog item needs to be refined before it can be prioritized, realized, and delivered [5, 6]. Business value is a broad concept that can be defined differently depending on the context and perspective, making refinement an essential step in agile. Not involving stakeholders will cause problems for the ASD team and lead to conflicts which can in turn lead to not delivering the expected solution and distraction from the final goal. Refinement is used within value-based software engineering (VBSE) [14,15,16], agile software development (ASD) [17, 18], and information system development (ISD) [12]. Biffl et al. [14] mention that benefits from value are not only monetary, but can also be economic, social, utilitarian, aesthetic, or ethical [14]. Salleh et al. [15] found in their systematic mapping study of 143 VBSE studies that there is no commonly accepted definition of value. 85% of VBSE studies refer to the value definition of Biffl et al. [14] and Boehm [16]: ‘VBSE brings such value considerations to the foreground so that software engineering decisions at all levels can be optimized to meet or reconcile explicit objectives of the involved stakeholders, from marketing staff and business analysts to developers, architects, and quality experts, and from process and measurement experts to project managers and executives.’

Chakraborty describes three important aspects of refinement: knowledge transfer, trust, and mental models/cognition [19]. Knowledge transfer is to overcome conflicting interpretations. This can be achieved by a continuous dialogue between the different stakeholders [19]. Trust can be increased by exchanging different points of view between stakeholders. The purpose of a shared mental model is to create a common understanding between product owner, stakeholders that request value, and the ASD team [19]. Deemer et al. stated that refinement contributes to preparing for the future and is part of continuous product development [20]. Palmer mentioned that the product owner has an explicit role in refinement [21]. Masood et al. [22] identified that refinement is done collectively with the entire team, but sometimes only between the product owner and the team lead.

Validation

It is important to check if the team is building the right product; this is called validation [7, 14, 15]. To make sure stakeholders get the requested value, it is important to validate before, during, and after realization. Within Scrum the sprint review is the validation event which is done amongst the stakeholders, product owner, and the ASD team [5]. Scrum does not prescribe how a sprint review is carried out, or which methods and techniques to use for review and validation [5].

Measurement

Research has been carried out on the formal measurement of business value delivery. Kupiainen et al. [23], Alahyari [18], Salleh et al. [24], and Sambinelli et al. [25] conclude that there is a lack of business value measurement methods and metrics within the area of agile software development. Kristinsdottir et al. [26], Dingsøyr et al. [1], and Huijgens et al. [27] conclude that measuring the actual business value delivered is difficult. Some researchers have made proposals of how to measure business value delivery; Hannay et al. define benefit points [28], and Hartmann et al. present Net Present Value (NPV), Internal Rate of Return (IRR), and Return on Investment (ROI) [29].

3 Research Method

To answer our research questions we conducted semi-structured in-depth interviews [30, 31] with 38 product owners in the Netherlands. The product owners can be differentiated in terms of their accountability for business, product, and technical decisions [32]. We used semi-structured interviews to collect data, uncover unexpected perspectives, elaborate on used practices, and collect insights on how product owners operationalize business value. We developed an interview protocol [31] to ensure consistency across the interviews.

The interview protocol contains four sections: 1) introduction, 2) opening question, 3) questions related to the four research questions, and 4) wrap-up. The introduction contains a very short explanation of the research in order to influence the participants as little as possible. During the introduction it was explicitly mentioned that the interviews are anonymous, and explicit approval was requested to record the interview. We kept our content-related questions to a minimum to allow the participants to give non-biased input as much as possible. We asked the participants the four research questions and requested examples, where available. The interview protocol was validated during the first interview and updated accordingly. In the wrap-up we asked the participants if they wanted to add anything that had not been touched upon. Furthermore we explained the follow-up steps after the interview. The interviews were conducted online (video and audio) via MS Teams, using the recording and automated transcription features. The interviews lasted between 33 and 81 min (average duration was 62 min). The interviews were conducted in Dutch. All transcriptions, coding, initial theme grouping, and determination of results was also done in Dutch, the native language of all participants and researchers. The translation of the results from Dutch to English was done while writing this paper. All interviews were fully transcribed, analyzed, and coded by the first researcher using reflective thematic analysis to determine patterns and construct themes [33]. The analysis was cross-checked by the second researcher. To minimize the risk of incorrect interpretations, a report was also created of each interview and sent to each participant for confirmation.

To address our research questions, we limited the participants to Dutch native-speaking product owners only. This was to ensure that we could exclude country-specific differences between the product owners. Speaking in the native language of the participants ensures that all nuances and details are covered, and limitations of participants expressing themselves in a non-native language can be excluded [34]. The validity consequences of this decision are discussed in Sect. 5.

We used a snowball sampling approach [35] in our personal network to ask connections to introduce us to 1, 2, or 3 product owners in their organization that we could interview. This resulted in 38 product owners from 17 organizations, across 10 different industries [36] (Table 1). The product owner experience of the participants can be found in Table 2.

Table 1. Overview of participating organizations and participants
Full size table
Table 2. Number of years of product owner experience
Full size table

4 Results

This section presents the results of how product owners (POs) operationalize business value in practice. This section is structured according to the four research questions.

4.1 RQ1: How Do Product Owners Determine the Most Valuable Backlog Items?

In the interviews we found that 26.5% of product owners use a kind of structured method, 47% use individual prioritization, and 26.5% do not use any prioritization method at all (Table 3).

Table 3. Categorization of prioritization
Full size table

We determined two structured methods: Weighted Shortest Job First (WSJF) [6] and Desirability Viability Feasibility (DVF) [42].

Own structured methods are either variants of WSJF or methods that have been developed completely from scratch within the organization. An example provided by ID05 in the ‘Own structured method’ category was developed within the organization and uses five criteria to score an epic or feature. An epic and feature receive points for each criteria, see Table 4. The sum of all the five criteria determines the priority of an epic or feature (<10 is priority 5; 10 <  = and < 15 is priority 4; 15 <  = and < 20 is priority 3; 20 <  = and < 30 is priority 2; > 30 is priority 1).

Table 4. Practice provided by ID05: prioritization score table
Full size table

For the five participants that use an individual list with criteria, we are not able to objectively determine any shared criteria.

4.2 RQ2: How Do Product Owners Refine Business Value?

Based on what the product owners mentioned, we determined five different refinement approaches. The main differences between these approaches is which stakeholders the product owner involves in refinement, and when the knowledge is transferred to the whole ASD team. Knowledge transfer is explaining the expected value by the stakeholders to the ASD team to create a shared mental model. Based on this we determine five different approaches:

  1. 1.

    Product owners refine with a sub-group of the ASD team.

  2. 2.

    Product owners refine with a sub-group of the ASD team and others.

  3. 3.

    Product owners refine on their own.

  4. 4.

    Product owners refine with a group of specialists that are not ASD team members.

  5. 5.

    Product owners refine directly with the whole ASD team.

We counted the number of times an approach was mentioned by the product owner, see Table 5. We conclude that 79% refine before the whole ASD team is involved. 21% directly involve the whole ASD team in the refinement. We also conclude that 65% of the product owners involve the whole or a few members of the ASD team in the refinement.

Table 5. Used refinement approaches
Full size table

ID01 uses a value model to create a shared understanding of what the stakeholder requesting the value expects. This model consists of four components: 1) main hypothesis, 2) sub hypothesis, 3) behavior (actions, thinking, feeling), and 4) metrics. Figure 1 contains an example. It starts with the hypothesis, which explains which stakeholder requires the business value and what is expected. The hypothesis still needs to be validated. It also describes the expected behaviors from the stakeholder that will use the delivered business value. It also defines how to measure the hypothesis.

Fig. 1.
figure 1

Practical example of a value model

Full size image

4.3 RQ3: How Do Product Owners Validate Business Value Delivery?

We found that the interviewed product owners use a variety of 17 different validation methods: panel of stakeholders (ID06, ID09, ID13, ID26, ID31, ID36, ID38), paid panel of stakeholders (ID25), technical validation (ID05, ID06, ID09, ID10, ID19, ID24, ID34), wireframes (ID06, ID25), employee surveys (ID09), automatic testing (ID12), prototyping (ID01, ID13), roadshows (ID13), user experience days (ID15), EPIC slides (ID17), mockups (ID20), AB testing (ID22, ID24), UX laboratory (ID22, ID24), guerilla testing (ID22, ID24), Google Analytics (ID26), integration testing (ID28), and customer visits (ID23).

We found that 84% of the product owners validate value and 16% do not validate value, see Table 6.

Table 6. Validate or not
Full size table

One product owner mentioned using five different validation approaches (ID01):

  1. 1)

    They validate the hypothesis, determined when making the value concrete, via interviews with internal users and customers.

  2. 2)

    During the design sprint they validate the prototype (also mentioned by ID15).

  3. 3)

    The ASD team carries out a test with a few customers or internal users.

  4. 4)

    During realization they make use of an ambassador group of internal users.

  5. 5)

    They validate with customers after putting the solution into production.

4.4 RQ4: How Do Product Owners Measure Business Value Delivery?

We found that 24% of the product owners state that they use metrics to measure business value, whereas 76% of the product owners state that they do not use metrics to measure value, see Table 7.

Table 7. Measuring business value
Full size table

We found that the 24% that use metrics measure these after value delivery. Some of them made estimates upfront during refinement and validation. The product owners that do use metrics to measure value mentioned some examples of how they measure business value delivery. The metrics used by ID01 are mentioned in Fig. 1 above.

Fig. 2.
figure 2

Practical example provided by ID25: value monitor

Full size image

ID25 developed a value monitor, see Fig. 2. The monitor contains five perspectives of value: internal customer, external customer, contribution to correct maintenance, innovation, and to the value the department delivers. The score is a scale from zero to five, whereby zero is no contribution and five is a significant contribution. This monitor is filled in by the product owner and the ASD team based on feedback from their stakeholders.

4.5 Main Question: How Do Product Owners Operationalize Business Value Delivery with Their Agile Software Development Teams?

In Table 8 we provide an overview of the number of activities used by the interviewed product owners. The four activities of operationalizing business value are: 1) determining most valuable backlog items, 2) refining expected business value, 3) validating business value, and 4) measuring delivered business value. We found that 4 of the 38 product owners use all 4 activities to operationalize business value. The majority of the product owners (55%) use only 2 activities to operationalize business value. We found that all product owners refine business value. We found that 84% validate business value. We found that just 26% use a structured method to prioritize backlog items based on business value. We also found that just 24% of the product owners use metrics to measure the delivered business value.

Table 8. Number of operationalizing business value activities used by the 38 product owners
Full size table

We observed differences between organizations but also between different product owners in the same organization.

5 Validity Threats

To evaluate the quality of our research, we reflect on the threats to the four validity dimensions: construct validity, internal validity, external validity, and reliability [30, 31, 43]. We mainly followed the validity guidelines from Runeson and Höst [43].

Construct validity concerns to what extent the operational measures being studied truly represent what the researcher has in mind and what is investigated according to the research questions [43]. We created an interview protocol containing all questions and explanations given to the participant. The interview protocol ensured that the same questions were asked and a consistent explanation was given to the participants each time. We used the answers and examples provided by the participant faithfully. When a participant, for example, stated that they use Scrum, we did not further investigate or question to which extent Scrum [5] is correctly implemented.

Internal validity concerns causal relations between investigated factors. Our research explored how product owners operationalize business value and does not aim to associate relationships [43]. All four sub-questions were analyzed separately. No causal relationships between the sub-questions were analyzed.

External validity is concerned with the extent to which our conclusions are valid outside the population participating in this research [43]. The external validity of our research is limited. Firstly, an individual product owner does not represent the other product owners in that organization. We attempt to mitigate this by selecting product owners from different industries and agile experience of the organization. Secondly, the included organizations are not representative of other organizations (also not within the same industry). Thirdly, the interviews are only held with product owners in the Netherlands, and are therefore not representative of other countries in Europe or the world. This is confirmed by the research differences with Polish product owners on prioritization [10]. Finally, participants have been selected by approaching product owners through the personal network of the researchers. This resulted in 38 product owners from 17 organizations, within 10 different areas of industry.

Reliability refers to the extent to which data and the analysis are dependent on the specific researchers [43]. To overcome cultural differences [44] and misinterpretations [34] of meaning of words, we decided to narrow our research to organizations from the Netherlands and native Dutch-speaking product owners. All questions and interviews were held in Dutch. We created an interview protocol based on the research questions. All interviews were conducted online (video and audio) via MS Teams, using the recording and automated transcription features of MS Teams. The first researcher conducted the reflective thematic analysis, which was cross-checked by the second researcher. To minimize the risk of incorrect interpretations, a report was also created of each interview and sent to each participant for confirmation. These approved interview reports were used to determine the results. All participants have confirmed the content of their report.

6 Discussing Research Questions and Future Research

RQ1: How Do Product Owners Determine the Most Valuable Backlog Items?

We found three main categories of prioritization: structured method (26.5%), individual prioritization (47%), and not using any prioritization (26.5%). Kukreja et al. [45] carried out research in 2011 and concluded that various ad hoc prioritization techniques are used and that there is a need for structured methods. We argue that a structured prioritization method or framework will contribute to the effectiveness of delivering business value, confirmed by Hujainah et al. [9] and Jarzębowicz et al. [10] and Kantola et al. [46], for product owner teams. We argue that these low figures for using a structured method and only three methods can be explained by the fact that Scrum does not prescribe any prioritization method, and SAFe recommends only one method: WSJF [6]. We also conclude that methods identified by Hujainah et al. [9] do not appear to find their way to product owners.

We suggest carrying out further empirical research into which structured methods can support product owners to optimize business value delivery.

RQ2: How Do Product Owners Refine Business Value?

We found that all 38 participants used refinement methods. The results further confirm the work of Palmer [21], where all product owners mentioned that they carry out refinement. Furthermore we have identified five different approaches to how product owners refine value. The advantage of approaches 1, 2, and 5 in relation to approaches 3 and 4 is that the understanding of business value by the ASD team members is likely to be higher, because the whole ASD team, or certain members thereof, are participating in building up the mental model, and in knowledge transfer from the stakeholders that request the business value. We found four approaches where 79% refine the business value before the whole ASD team gets involved. And one approach (21%) that directly involves the whole ASD team. This finding is interesting as it contradicts the agile approach, which stresses the importance that ASD team members are in direct contact with their stakeholders (fourth principle of the agile manifesto [2]).

Future research is required to determine what the pros and cons are of these approaches, and if these approaches can contribute to more involvement of business stakeholders as suggested by Alami et al. [47] and Hoda et al. [48].

RQ3: How Do Product Owners Validate Business Value Delivery?

We found that there is no one common way of carrying out validation, and that a variety of different methods are used. Scrum [5] mentions the sprint review but does not prescribe how to perform this. Salleh et al. [15] identified nine studies in the VBSE principles & practices VB verification and validation. We argue that validation methods appear to be context-specific and that product owners should therefore have more knowledge of when and how to use which specific validation method.

The extent to which product owners are aware of the broad availability of such methods, their applications, strengths and weaknesses, and how product owners are trained to apply them appropriately, is a topic for future research.

RQ4: How Do Product Owners Measure Business Value Delivery?

We see that just 24% of the product owners explicitly measure value. Our research confirms the conclusions of Kristinsdottir et al. [26], Dingsøyr et al. [1], and Huijgens et al. [27] that measuring delivered business value is difficult.

We suggest investigating if a measurement system can be developed for measuring business value delivery, taking into consideration, for example, the 47 value propositions mentioned by Rodríguez et al. [11].

We suggest research to investigate how measuring business value can contribute to increasing the delivery of business value, prove that the delivered value relates to the investment, and on how to select and use the most suitable metrics.

Main Research Question: How Do Product Owners Operationalize Business Value Delivery with Their Agile Software Development Teams?

We found that only 10.5% of the product owners are using all four activities of operationalizing business value, 24% are using three activities, the majority (55%) are using only two activities, and 10.5% use just one activity. There could be an explanation for these low figures because the ‘ideal product owner’ does not exist. Kadenic et al. [32] come to the conclusion that the ideal product owner does not exist or that the responsibility of the product owner cannot be carried out by one person. Based on these findings and research sub-questions, we argue that operationalizing business value delivery by product owners in practice requires improvement. Organizations should invest in further development of the skills and capabilities of the product owner, which should contribute to increasing the level of business value delivery. This is also supported by the research of Kantola et al. [46].

7 Conclusion

The extent to which product owners operationalize business value delivery differs considerably in practice. These differences are not only observed between organizations but even between different product owners in the same organization. In order to improve business value delivery from the product owner perspective, there is a clear need for better structure and guidance in operationalizing business value delivery in daily practice. This is also confirmed by the lack of structured methods or frameworks used by these 38 product owners. Some local practices have been found in the interviews, yet only a few product owners operationalized business value delivery in a repeatable and structured way.

Furthermore we conclude that, in practice, operationalizing business value delivery is not in place. Only four product owners are applying all four activities to operationalize business value delivery. We found large differences in perspectives among product owners between validating and measuring value. The majority (84%) of the product owners in this research do claim to validate value, yet measuring value is done only by a few of them (24%). A structured method to operationalize business value could as such bridge this gap between perceived validation and collecting quantitative evidence.

In general we conclude that there are several opportunities to take product ownership to the next level in operationalizing the delivery of business value in practice.