Keywords

1 Introduction

Agile software development methods were originally designed for small and co-located teams. The realized benefits in small organizations led to an increased interest in agile across large-scale organizations [1]. Transformation to agile is not an easy undertaking; several studies have reported significant challenges while adopting agile in large-scale settings, e.g., change resistance, coordination challenges in multi-team environment, and challenges in involving non-development units [1]. Agile adoption is more than just implementing practices; it is about changing the mindset, and culture [2]. Several agile scaling frameworks, e.g., Scaled Agile Framework (SAFe) [3], Large Scale Scrum (LeSS) [4], Disciplined Agile Delivery (DAD) [5] were designed by practitioners and consultants to support scaling of agile to large organizations. Out of all scaling frameworks, SAFe has been the most popular according to the most recent State of Agile survey by VersionOne, with 35% of the respondent organizations reporting its usage [6].

The popularity of SAFe and successful marketing of its benefits by the Scaled Agile Inc. has encouraged companies to take SAFe into use [7]. Over one hundred companies have reported about their SAFe usage via case studies and short stories [8]. According to the official SAFe website [3], 70% of Fortune 100 companies have certified SAFe professionals, and 700,000 practitioners have been trained in SAFe. Despite the popularity of SAFe in industry, scientific research exploring SAFe usage is still limited; nevertheless, there is a slight increase in scientific studies published after 2018. However, the majority of reported experiences on SAFe still comes from grey literature [7], most of which is published on the SAFe official website, which may lead to biased information. The SAFe adopters have reported several benefits [7]. However, we do not have much research-based evidence of these benefits. Therefore, it would be essential to identify whether the SAFe framework can mitigate the challenges of scaling agile, bring the promised benefits, and determine whether it brings in new challenges [9].

A few quantitative studies have already reported the benefits and challenges of SAFe usage [10,11,12]. However, we need more quantitative studies to capture the state-of-practice of SAFe adoption, as the limitations of the existing surveys prevent generalization, e.g., in [10], out of 111 respondents, only 5% were pure SAFe users, and [11] studied only three organizations. As many researchers have expressed the need for better understanding the SAFe usage and adoption [13, 14], in this paper, we investigate the state-of-practice of SAFe adoption by conducting an empirical survey of companies that have taken SAFe into use. We report the contextual factors of companies, as well as the benefits and challenges of SAFe usage.

The remainder of this paper is structured as follows. In Sect. 2, we present the background and related work of our paper. Section 3 describes the underlying research method of this paper. Section 2 presents our results and provides a discussion of our main findings. Section 5 concludes our study with a summary of our results and remarks on future research.

2 Background and Related Work

This section gives an overview of SAFe and presents the previous empirical studies on SAFe.

2.1 SAFe

Dean Leffingwell established SAFe in 2011. The latest version, 5.0, was released in 2020 [3]. SAFe incorporates practices from agile and lean [3]. It has four different configurations: Essential, Large Solution, Portfolio, and Full SAFe. Each configuration has set of practices, artifacts, and roles to deliver solutions to the end user. SAFe has four core values: built-in quality, transparency, alignment, and program execution [15].

SAFe claims that the most common benefits of its adoption are: employee engagement, productivity, time to market, and quality [3]. Challenges of implementing SAFe are not mentioned.

2.2 Previous Studies

In this section, we present the benefits and challenges of SAFe adoption reported in the previous studies.

Putta et al. [7] conducted a multivocal literature review (MLR) and reported a total of 23 benefits and 15 challenges of SAFe adoption. The most common benefits were: transparency, alignment, and quality. The most often mentioned challenges were: resistance to change, moving away from agile, and controversies within the framework. The study’s limitation is the use of grey literature from the SAFe website that might lead to the results being biased towards the benefits of SAFe.

Laanti and Kettunen [10] conducted an empirical survey on SAFe adoptions in Finland. They analyzed data from 111 respondents. As the most significant benefits of SAFe they reported: transparency, co-operation, and common cadence. The most commonly mentioned challenges were: old mindset and culture, the model not fitting correctly to own organization, and missing fluency when using the model. The limitation is that only 5% of the respondents were pure SAFe users.

Gustavsson and Bergkvist [11] surveyed SAFe in three different organizations: automotive, financial, and public sector. They reported increased visibility, overview, and transparency as the most common benefits and the lack of productivity, focus, and efficiency as the most common drawbacks of SAFe adoption. As the study had only three organizations participating, it makes it hard to transfer the results to other organizational settings.

Salikhov et al. [12], surveyed 16 organizations that had adopted SAFe and received answers from 21 respondents. Their preliminary results indicate improved productivity, better handling of dependencies, improved coordination between levels, and better vision of the big picture as the most often mentioned benefits. The most common drawbacks include: requires more resources, complex structure, lack of autonomy, and decreased productivity.

The existing surveys are few, and they are limited to selected locations and organizations, which reduces their external validity. In this paper, we present the benefits and challenges of SAFe by conducting an empirical survey with respondents from various geographical locations, domains, roles, and differing length of experience with SAFe, which helps to understand SAFe usage more broadly in the industry and improves the external validity of the findings.

3 Research Methodology

This section describes the research design by presenting the research questions, survey design, data collection, and data analysis.

3.1 Research Questions

We aim to answer the following research questions:

  • RQ1:What benefits are realized after the adoption of SAFe?

  • RQ2: What challenges are witnessed during and after the adoption of SAFe?

3.2 Survey Design

In order to answer the research questions, we created a survey following the guidelines suggested by Linåker et al. [16]. The survey questionnaire consisted of six sections with a total of 25 questionsFootnote 1. In the first section, we presented our research goals and information on who should answer the survey (only practitioners having experience in one or several of the scaling frameworks were asked to answer the survey). The second section included questions on the organizations’ transformation background, such as how long they had used a specific framework. In the subsequent three sections, we asked the participants to assess their agreement regarding their own organization on lists of reasons, benefits, and challenges, which were compiled based on three previous studies on agile and large-scale agile development [17,18,19]. Additionally, in each of the three sections described above, we included an open-ended question for the respondents to add other reasons, benefits, or challenges experienced/witnessed respectively to reduce the anchoring effect. In the fifth section, we captured the participants’ background information, such as their company’s domain, respondent’s primary role in the organization, and the location of the organization. The last section provided closing remarks on the survey and a thank you message.

Survey Validation. The questionnaire was first carefully reviewed by first three authors. Then, we asked for comments from an academic subject matter expert. Next, two survey experts helped to make sure that the questionnaire adhered to the best practices of survey research. Finally, an industry expert on large-scale agile reviewed the questionnaire and suggested, e.g., reducing the questionnaire’s length. After incorporating all suggestions, we conducted a pilot survey with three respondents and asked for their feedback on the questionnaire’s length, understandability, and readability. After final modifications, the survey was ready to be submitted to the target audience.

Sampling and Target Audience. In this context, we used non-probabilistic convenience sampling, which involves “getting responses from the individuals who are willing and are available” [20]. The target audience for the survey included software professionals from various roles, e.g., developers, managers, coaches, who use agile scaling frameworks in their organizations.

Data Collection. The data collection took place between May and September 2019 using a third-party, online tool “LimeSurvey”Footnote 2. To reach our target population, we promoted the survey in: (1) conferences, (2) meetup groups, (3) social media groups, and (4) via personal networks.

We promoted the survey in three conferences: the 20th International Conference on Agile Software Development (XP 2019)Footnote 3, the 14th International Conference on Global Software Engineering (ICGSE 2019)Footnote 4, and the Agile 2019Footnote 5.

Two researchers promoted the survey at the XP 2019 conference. They spoke to people during the breaks and mailed the link to those interested in answering the survey. A link to the survey was distributed among all XP 2019 participants in a conference news post. At the ICGSE 2019 conference, two researchers promoted the survey in the same way as at the XP 2019. At the Agile 2019 conference, one researcher promoted the survey by handing out cards containing survey information and a QR code to the online survey. The survey link was also sent out to all participants via email in the daily conference news letters during the conference, as well as one week after the conference.

The survey link was also published on selected social media platforms, promoted at Agile Meetups, and distributed to professionals. In June 2019, the survey link was posted in the worldwide LinkedIn group “Lean and Agile Software Development”Footnote 6, which is the largest online community of Lean and Agile practitioners with more than 157.000 members from all over of the world. We promoted the survey in two Agile Meetups, where practitioners share their experiences, ideas, and knowledge on issues regarding agile software development. One was held in Helsinki, Finland, in August 2019 with 32 participants and one in Copenhagen, Denmark, in June 2019 with 30 participants. The survey leaflets, containing the link and QR code, were distributed to all Meetup participants. Finally, professionals from different organizations worldwide were approached via email, LinkedIn, and other social media channels and asked to fill in the survey. By snowballing of contacts we aimed to find new contacts, i.e., the personal networks of existing contacts were leveraged.

3.3 Data Analysis

We imported the collected survey data from LimeSurvey to the SPSS Statistics toolFootnote 7. We conducted a two-day workshop among all authors to clean the data, e.g., removed incomplete responses and agreed on how to conduct the statistical analysis.

Descriptive Statistics. We started the data analysis by running the basic descriptive statistics for contextual information, benefits, and challenges of SAFe, such as frequencies, to get an overview of the data and insights on how to proceed with inferential statistics. Then, we calculated the mean values for both benefits and challenges.

Inferential Statistics. To test the normality of data, we conducted the Kolmogorov-Smirnov test [21], which showed that our data had a non-normal distribution. Thus, we adopted non-parametric tests to conduct inferential statistics. We used the Mann-Whitney U [22] to compare the differences between two independent groups, e.g., duration of the agile scaling framework usage, when the dependent variable is either ordinal or interval/ratio, e.g., benefits and challenges. In the case of more than two groups, we used Krusal-Wallis H test [22].

4 Results and Discussion

In this section, we present an overview of contextual information of our respondents and answer our research questions on the benefits and challenges of adopting the SAFe framework, as well as compare to previous findings. Finally, we discuss the limitations of our study.

4.1 Overview of the Contextual Information

In total, we received 204 responses to our survey. 100 respondents had adopted SAFe as their primary framework that was predominantly used in their organization, while the rest of the respondents had adopted other scaling frameworks as their primary frameworks. We separated the data of SAFe respondents and analyzed them to answer our research questions. Next, we present the contextual and descriptive information of the SAFe respondents.

Geographic Distribution of Respondents. The respondents that adopted SAFe were distributed to all continents except South America. As shown in Fig. 1, the highest number of respondents were from the USA (24% of respondents), followed by Germany (16%), and Denmark (11%). The geographical distribution of our respondents matches the information on the SAFe web page, as most organizations using the SAFe framework are reported to be located in the USA [8]. Another plausible explanation that most of our respondents are from the USA is that we approached participants in conferences that took place in the USA (Agile 2019) and Canada (XP 2019, ICGSE 2019). The next highest response rates were from Germany and Denmark, as two authors were located in these countries leading to more responses from those locations.

Fig. 1.
figure 1

Geographical distribution of the respondents

Full size image

Roles of Respondents. The highest number of our respondents (see Fig. 2) had a process related role (57% of the respondents), such as Scrum Master or agile coach, followed by the management roles (26%), such as project and line managers. Roles from development team level (8%) included developers and team leaders and roles from product level included Product Owners and requirement engineers (8%).

Fig. 2.
figure 2

Roles of the respondents

Full size image

Duration of Usage. Most of our respondents had started their SAFe adoption either 1–2 years ago (40% of the respondents) or 3–5 years ago (40%), while 13% had less than one year of SAFe experience and only 7% had more than five years of experience in implementing SAFe (see Fig. 3).

Fig. 3.
figure 3

The number of years of experience since SAFe adoption

Full size image

Previously used Development Approaches. The majority of our respondents used plan-driven methods before adopting SAFe (52%) (see Fig. 4).

Fig. 4.
figure 4

Previously used software development approaches before SAFe adoption

Full size image

Industry Sector. Our respondents’ organizations that had adopted SAFe came from several different domains. As shown in Fig. 5, the highest percentage of respondents came from the financial sector (33% of the respondents), followed by the public sector (13%) and technology domain (12%). The distribution of the domains matches with the results from a prior MLR on SAFe [7], that indicated that many organizations adopting SAFe were from the finance and technology sectors.

Organizational Areas in which Framework was Applied. The inventors of SAFe have designed it to scale agile and lean practices to whole enterprises, indicating that the adoption of SAFe is not only limited to software development, but can also be used at higher organizational levels or in other organizational units, such as marketing or human resources [3]. However, our survey data shows that the majority of our respondents have adopted SAFe primarily in the IT (74 responsesFootnote 8), and product development (59 responses) areas (see Fig. 6).

Fig. 5.
figure 5

Industry sectors of the respondents’ organizations

Full size image
Fig. 6.
figure 6

Organizational areas in which SAFe has been adopted

Full size image

4.2 Benefits of Adopting SAFe

Based on the previously published survey [17], we identified eleven benefits that the adoption of agile entails (see Fig. 7). Our respondents were asked whether their organizations realized these benefits after adopting SAFe. To better understand the respondents’ agreement on the realized benefits of adopting SAFe, we calculated the mean values for each benefit. In Fig. 7 the benefits have been arranged from highest to lowest mean values.

Fig. 7.
figure 7

Agreement of the Benefits of SAFe, arranged according to the Mean Values

Full size image

Our respondents provided the highest level of agreement (sum of strongly agree and agree) that the adoption of SAFe has improved collaboration between agile teams (71% of the respondents), followed by the statements that the adoption of SAFe has resulted in improved dependency management between teams (68%) and improved transparency (66%). Our findings on the most realized benefits were also the most frequently cited benefits in previous SAFe surveys and literature reviews [7, 10, 11]. These benefits also align with the two core values of SAFe: alignment and transparency [15]. Furthermore, the other benefits we found were also identified in a prior MLR on SAFe [7].

According to the SAFe website, organizations using SAFe have reported that its adoption improved their software quality by 50% [3]. However, in our survey, software quality received the least agreement and had the lowest mean value, indicating that a big part of our respondents disagreed with the statement that the adoption of SAFe has improved the software quality. This finding also contradicts with one of the core values of SAFe: built-in quality [15]. However, we did not measure the actual increase or decrease of software quality in this survey. Thus, we believe that further research on actual measurements is needed. Other benefits claimed by SAFe are related to improved engagement, productivity, and time to market [3]. However, we did not investigate the first two benefits claimed by SAFe in our survey, as the survey was designed to capture general benefits that would be common to several frameworks. Nevertheless, we identified a faster time to market in our survey, with 50% of our respondents agreeing on this benefit after adopting SAFe.

Previous Development Methodology. A systematic mapping study of Uludağ et al. [23] showed that the most organizations transforming to large-scale agile development by adopting agile scaling frameworks, used before the transformation either plan-driven or traditional agile methods, such as Scrum. Thus, we wanted to determine whether previous experiences in agile methods resulted in better-realized benefits compared to those having previous experience in plan-driven methods. We conducted the Man-Whitney U test to identify a possible difference between those two groups. We took the previous development methods (plan-driven and agile) as the independent variable while the dependent variable was each of the benefits.

We did not identify any statistically significant difference between the two groups for any of the benefitsFootnote 9. However, mean ranks for all the benefits, except for to enable faster feedback, to have more frequent deliveries, and to have shorter time to market, were higher for agile organizations when compared to plan-driven organizations. This might indicate that the agile organizations experienced these benefits more due to their longer experience in agile in general. Previous studies have reported such benefits of using agile methods [6, 24, 25]. For many plan-driven organizations, the three benefits listed above might have been experienced only after using SAFe; therefore, respondents from plan-driven organizations had a greater agreement with these benefits when compared to agile organizations.

Duration of Usage. Typically, organizations should be able to see more benefits after a longer time from the adoption of agile or agile scaling framework. For example, a case study on SAFe adoption found that the second unit in the same organization adopting the framework was more successful than the first one [26]. Thus, we were interested in determining whether a longer time horizon of SAFe usage results in better-realized benefits than a shorter time horizon.

To test this, we conducted the Kruskal-Wallis H test by taking the years of experience of SAFe usage as the independent variable (less than 1 year, between 1–2 years, between 3–5 years, and more than 5 years) and the realized benefits of SAFe adoption as the dependent variable.

We found a statistically significant difference for the following benefits: to improve team autonomy (Kruskal-Wallis H = 10.49, p = 0.015), to have more frequent deliveries (Kruskal-Wallis H = 14.244, p = 0.003), to have shorter time to market (Kruskal-Wallis H = 12.028, p = 0.007), and to enable faster feedback (Kruskal-Wallis H = 11.407, p = 0.01) meaning that companies with longer experience of SAFe reported experiencing more of these benefits. We also observed that mean ranks for most of the benefits increased with an increase in SAFe adoption duration. These results seem to indicate that organizations may realize the full extent of the benefits of adopting SAFe only after a longer time horizon. Our results are in line with previous studies [24, 27]. For instance, Laanti et al. [24] concluded that years of experience with agile methods may positively influence their usefulness.

4.3 Challenges of Adopting SAFe

Figure 8 shows the results for the sixteen challenges identified in the literature, arranged from highest to lowest mean values.

The challenge that received the highest agreement (sum of agree and strongly agree) among the respondents was organizational politics (72% of the respondents) which also has the highest mean value. This challenge was also reported as a significant challenge during a SAFe transformation in a case study conducted in a financial organization [28]. The next most agreed challenges were difficulties in establishing an agile mindset (68%), change resistance (67%), and team formation challenges. As the majority of our respondents transformed to SAFe from plan-driven methods, the difficulties in establishing an agile mindset and overcoming change resistance are understandable. These aforesaid challenges were also frequently identified in large-scale agile endeavours [1, 6] as well as in SAFe adoptions [7, 28].

The challenge, framework does not help in resolving problems with dependencies between development teams (14%) received the least agreement among the respondents and the smallest mean value. While challenges related to dependency management between agile teams were quite commonly mentioned in large-scale agile endeavours [1, 6] and SAFe adoptions [7], by disagreeing this statement (69% disagreed) our respondents indicate that adopting SAFe might actually help in managing dependencies between agile teams. This is further conformed by our survey results on SAFe benefits, as the improved dependency management between teams received second highest agreement by our respondents.

Other challenges, such as difficulties in including non-development units (63% agreed), and difficulties in staffing new roles (56% agreed) have also been reported by an MLR on SAFe [7].

Fig. 8.
figure 8

Agreement of the Challenges of SAFe, arranged according to the Mean Values

Full size image

Previous Development Methodology. We tested whether the agreement on challenges faced differed between organizations using previously plan-driven methods and those using previously agile methods by conducting the Man-Whitney test.

We found statistically significant differencesFootnote 10 regarding the following challenges: uncertainty with respect to middle management’s role in agile (Mann-Whitney U = 102.5, p = 0.0046) and difficulties in getting the management buy-in (Mann-Whitney U = 88, p = 0.0018), meaning that previously plan-driven organizations agreed to these challenges more than agile organizations. These results match previous findings that plan-driven organizations struggle to clarify the new roles of managers in agile environments compared to organizations having already experience with agile methods [28]. Similarly, getting the management buy-in in plan-driven organizations is more complicated than in agile organizations due to power struggles and politics [28].

The mean rank for most of the challenges for respondents coming from plan-driven organizations was higher than for respondents from agile organizations, which indicates that previous agile experience may ease SAFe adoption. Only for the following challenges: the scaling framework brings additional work overhead, using the framework felt like moving away from agile, and missing guidance on architectural topics had a higher mean rank among the organizations that had already used agile, indicating that already agile organizations felt these more problematic than plan-driven organizations, which could be expected. For example, a case study on SAFe transformation from the Scrum model reported these same challenges and mentioned that long fixed increments of SAFe (e.g., a 8-week Program Increment cycle) may look like a step back to the plan-driven world for an organization that is already used to the fast agile planning cycle [26].

Duration of Usage. We tested whether the agreement on the challenges experienced differed with respect to how long time the respondent’s organization had been using SAFe. We conducted the Kruskal-Wallis H test to find out if there was a statistical difference between each of the challenges and the four groups of the duration of SAFe usage (less than 1 year, between 1–2 years, between 3–5 years, and more than 5 years).

We did not find statistically significant difference for any of the challenges among the four groups. While comparing the mean ranks, and we did not find any pattern of increase or decrease regarding the number of years since SAFe adoption. This finding is not surprising as we asked from the respondents whether they perceived the challenges when adopting the agile scaling framework, thus our data cannot answer to the question on whether they still experience the same problems.

4.4 Threats to Validity

Although we employed a rigorous survey design and paid attention to data collection and analysis, there are limitations that are discussed next and organized as suggested by Wohlin et al. [29].

Internal Validity. This threat concerns factors that can influence the relationship between the research process and the obtained results, e.g., respondent bias. We mitigated the respondent bias by collecting data from reliable sources: most of the responses came from people we met during the conferences and Meetups, and we knew they were using the agile scaling frameworks, which helped us to avoid unreliable or unauthentic responses. As the questionnaire consisted of separate sections investigating the benefits and challenges of adopting agile scaling frameworks, respondents could not overemphasize the positive elements of the SAFe adoption.

External Validity. This threat is related to the generalizability of our results. We counteracted this limitation by having respondents with different roles, working in different domains, coming from various countries, and having different length of SAFe experience. This helped us to improve to external validity of our survey when compared to previous quantitative studies (e.g., [10, 11]).

Construct Validity. This treat concerns whether the questions asked in the questionnaire represent the attributes being measured. We formulated the survey statements on the benefits and challenges based on earlier findings in the literature. However, as we had to limit the questionnaire length, we could not include all the benefits and challenges we identified from the literature. Therefore, were not able to fully address this threat, which is a prevalent problem in survey research [16]. We validated the questionnaire with a domain expert, survey experts, and tested it by conducting a pilot study. This helped to make the questionnaire as clear and understandable to the respondents as possible.

Conclusion Validity. This threat is concerned with the ability to draw the right conclusion from the collected data. The survey data was mostly Likert data, and we conducted appropriate non-parametric tests for identifying differences between independent groups. We also compared the results with the existing literature for validating our results.

5 Conclusions and Future Work

This study provided empirical evidence on the adoption of SAFe in industry. We analyzed data from 100 practitioners using SAFe as their primary scaling framework. Our results show that the three topmost realized benefits of adopting SAFe are: improved collaboration between teams, improved dependency management between teams, and improved transparency. Improved software quality is the least agreed benefit, even though the SAFe founders claim it to be a common benefit from SAFe. It is important to note that we did not measure the actual increase or decrease of the benefits. However, our contribution to practitioners is to provide objective information on the SAFe benefits as experienced by SAFe user organizations. We encourage the researchers to collect actual metrics used to quantify the benefits of SAFe usage.

Our results further revealed that the most common challenges of adopting SAFe were organizational politics, difficulties in establishing an agile mindset, change resistance, and team formation challenges. The last two challenges were also frequently reported in previously conducted reviews and surveys in large-scale agile development. As the present literature on large-scale agile development is mostly problem-centric [23], we encourage researchers to investigate what types of solutions have been adopted by organizations to address the challenges witnessed during transformation.