6.1 Introduction

Globalization, digitization, new ways of working, knowledge culture, mobility, and connectivity are so-called “Megatrends” that are changing the world and thus also how companies and organizations operate (Horx, 2020). These trends and constant changes in the environment emphasize the importance of innovation as an essential part of long-term survival for every company in today’s VUCA World (Jöhri, 2020). The acronym VUCA stands for V = Volatility; U = Uncertainty; C = Complexity; A = Ambiguity and was first introduced in 1987 by Warren Bennis and Burt Nanus (Wikipedia, 2020). The need to constantly adapt and innovate is especially true for small countries like Switzerland (Jöhri, 2020). Continuous adaptation understood as agility is crucial as stagnation implies “going backwards” (Jöhri, 2020). Consequently, organizations must seek out solutions in order to survive and stay competitive. Carroll (2019) specifically identified agile management as one possible way of dealing with the challenges in today’s fast-moving economy. More than ever companies need to refine and innovate. Agile methods are often introduced as a means to support continuous innovations and refinements (Wilson & Doz, 2011, pp. 6–26). As innovation play a major role and are crucial for the resilience and success of every organization, agile methods are introduced to support a decentralized and faster moving innovation processes. However, recent research has pointed out that despite agile methods increasing speed and efficiency, team dynamics are equally important (Baldauf, 2020). And others have highlighted that every change, including the implementation of agile methods, can involve unintended consequences. This begs for a closer examination of the empirical consequences and challenges of implementing agile methods to improve the innovation process and long-term organizational resilience. The chapter will address this research opportunity and present empirical findings that help the reader better understand the promise and potential pitfalls of agile methods. Before we present our findings, we will provide the reader with a short definition of the key terms and existing research.

6.2 Theoretical Background

Schumpeter (1934, p. 13) known as the father of innovation, defined innovation as “the commercial or industrial application of something new – a new product, process, or method of production; a new market or source of supply; a new form of commercial, business, or financial organization”. Since its introduction in 1934, the focus on innovation has become one of the key success factors for every organization. Furthermore, Kahn (2018, pp. 453–460) defined innovation as “an outcome, a process, and a mindset”. Yet, despite having many creative ideas, many organizations don’t successfully implement innovation processes (Aslam et al., 2020, pp. 1–24). Aslam et al. (2020, pp. 1–24) investigated this phenomenon and identified the absence of effective innovation management frameworks as the key barrier to innovation. Agility is seen by some as offering such a framework and is therefore introduced next.

6.2.1 Agility

According to Ravichandran (2018, p. 25) agility is a “firm’s capacity to respond with speed to environmental changes and opportunities”. It should be noted that to achieve operational agility, the relation among the following three dimensions seems important: customer responsiveness, operational flexibility and strategic flexibility (Ravichandran, 2018, p. 25). Customer responsiveness is defined as an ability to identify customer needs and preferences and hence to respond quickly with a product or a service; operational flexibility is the ability to optimize processes and achieve improvements in the speed of product development, delivery and logistics processes; finally, strategic flexibility is the ability to locate and access new markets and redefine the scope of business (Ravichandran, 2018, p. 25). Furthermore, agility has been indicated as a suitable framework for innovation in today’s economy because of its positive impact on firm performance (Ravichandran, 2018, pp. 22–42). Some of the most innovative companies such as Apple, Microsoft, Amazon, Google employ agile methods and trust agile teams to create new competitive advantages (Rigby et al., 2018, pp. 1–10). Many organizations try to follow their model and attempt to become more agile. According to Lee (2020, pp. 85–87) the road to an agile organization involves an organizational transformation including a new organizational culture that breaks with the traditional vertical organizational structure and shifts responsibility and decision power to small self-responsible teams (Lee, 2020, pp. 85–87). Moreover, agile methods and its utilization by the teams within an agile working environment are of great importance for operating in an agile manner. Consequently, agile methods support the daily work of the agile teams and are essential for the innovation. Although some organizations are born agile and use agile methods from the beginning of their operations, most of them must go through an agile transformation in order to operate in an agile manner (McKinsey, 2019, p. 3). Below, we specify the implications and meanings of an agile transformation as key for becoming more agile.

6.2.1.1 Agile Transformation

Agile organizations differ considerably from traditional organizations and transition to an agile organization usually requires a fundamental, lengthy shift (McKinsey, 2019, pp. 1–10). According to McKinsey (2019), the process of agile transformation affects every part of the organization: people, processes, strategy, structure and technology. Therefore, it is necessary to be comprehensive and iterative in the process of agile transformation, since not everything can be planned in advance (McKinsey, 2019, p. 2). Although organizations have different approaches to agile transformation, some components such as aspiration, design and agile pilots are the same for every organization (McKinsey, 2019, pp. 2–3). These elements can be seen below as part of a two phase iterative approach to transformation (Fig. 6.1). As stated above the process of agile transformation requires sufficient time and efforts in order to overcome the challenges. Especially problematic are barriers that revolve around employees’ knowledge of agility, their attitudes towards it (their mindsets which are generally negative towards any kind of change) and cultural issues (Gandomani & Nafchi, 2016, pp. 257–266).

Fig. 6.1
figure 1

Two components make up an iterative approach that requires the organizations to continuously test, learn, and course correct. (McKinsey, 2019, p. 3)

Full size image

One of the crucial factors that determines the success of agile transformation is the self-organized team. Lindsjørn, Sjøberg, Dingsøyr, Bergersen, and Dybå (2016) investigated the quality of agile teamwork and its impact on team achievement and knowledge development. Moreover, they also compared the performance of agile teams with teams not using agile methodology. Surprisingly, they concluded that there is no significant difference between the quality of work within agile teams and other teams (Lindsjørn et al., 2016, pp. 274–286). Such results beg for a closer examination of the factors that help teams improve their innovation via agile methods.

6.2.2 Agility and Innovation

Innovation is no longer a choice, but a necessity. Companies must constantly innovate to survive (Brand et al., 2021, pp. 157–187). Because the framework of agility places great emphasis on reacting quickly, flexibly and iteratively it has often been identified as the ideal framework for constant innovation (Fisher, 2019). Moreover agile methods give credence to the customer and their feedback, which strengthens the process of innovation (Fisher, 2019) and authors such as Denning (2017) have shown the effectiveness of agility and strategic management in encouraging innovation. Besides the impact of agility on innovation in the software industry, studies have shown the positive impact of agility on the process, product and business model innovation (Bouncken et al., 2019, pp. 1–14).

The agile manifesto places emphasis on the importance of individuals and interactions over tools and processes (Beck et al., 2001). Grass, Backmann, and Hoegl (2020, pp. 324–351) investigated the role of agile team members, leaders and their interactions. Their study found that empowerment was crucial in the process of adapting to the new agile working environment (Grass et al., 2020, pp. 324–351). They also identified the importance of external factors such as customers and organizational environment to team adaptability (Grass et al., 2020, pp. 336–343). As a result, the authors created a model of continuous agile team innovation in which adaptability is the most important capability for fostering innovations (Grass et al., 2020, p. 337). This model is presented in Fig. 6.2.

Fig. 6.2
figure 2

Model of the Continuous Agile Team Innovation Process (CATIP). (Grass et al., 2020, p. 337)

Full size image

Research by Vishnubhotla, Mendes, and Lundberg (2020, pp. 1–18) on agile team climate suggests that positive team atmosphere depends on the personality of the team members, above all on their agreeableness – the ability to get along easily with other team members. Something that seems reasonable also for teams who are not explicitly working with agile methods. Furthermore, they found that there is a positive correlation between team members’ personalities such as open-mindedness and the willingness to experience in the process of supporting innovations (Vishnubhotla et al., 2020, pp. 1–18).

To sum up, the current state of research identifies agility as a suitable framework for continuous innovation. However, there are significant challenges on the road to fully adopting agile methods. This study therefore set out to explore the practical effects of agile methods on innovation by asking the following research question: What are the practical experiences of working with agile methods in innovation? Before we report the findings, we briefly present the research methodology.

6.3 Research Methodology

To answer the research question, we conducted qualitative interviews. Qualitative interviews systematically collect data through conversation based on an interview guideline, ensuring that all interviews address the same question while also leaving freedom to adapt the interview to the situation (Bryman & Bell, 2011).

A total of 15 interviews with an average duration of 1 hour were conducted with employees of a Swiss telecom company that went through an agile transformation 5 years ago. The specific selection of the interview partners was based on their availability, team affiliation and on the indications of who could provide additional information concerning the main topics. An overview of the selected interviewees, their function and department is presented in Table 6.1.

Table 6.1 Overview of the data collection. (Own illustration)
Full size table

Because of the COVID-19 pandemic, all interviews were conducted via online communication channels such as Skype, Zoom and Microsoft Teams. The interviews were carried out in German or English, depending on the interviewee’s preference. All interviews were recorded electronically and transcribed verbatim. Following the transcription of the interviews, a systematic content analysis of the data was carried out, identifying themes and codes to answer the research question with a focus on advantages and challenges and the tensions that arose from their juxtaposition.

6.4 Results

The results of this research project are presented with a focus on the research question. Key topics or findings are directly supported by statements from the interviews and additional aspects are described in the discussion.

6.4.1 Agility and Innovation: A Necessity or Support Function?

To better identify the effects of agility management on innovation, we asked the respondents to share their opinion on whether agility is a necessity or a supporting factor for innovation. The majority of respondents saw in agility a supporting function for innovation (Fig. 6.3). Moreover, it was indicated that to focus on agility alone is not sufficient and that some additional factors should be considered in the innovation process.

Fig. 6.3
figure 3

Agility as a support function for innovation. (Own illustration)

Full size image

As presented in Fig. 6.3, agile methods can support innovation, but are not a necessity for innovation. Furthermore, agility was seen by the interviewees as part of the zeitgeist; a fashion or popular trend that does not guarantee more innovation. Moreover, some noted that agility and agile methods are not having an impact on the innovation, as innovation can be encouraged with other methods such as the waterfall approach. Consequently, it was indicated that innovation needs the right people who are motivated, curious, and open to new things. In addition, the notion of “free spaces” for innovation including time to think and experiment were mentioned as essential for innovation and could be considered as a necessity, since innovation cannot happen overnight. Besides the free spaces for innovation, the type of innovation was highlighted as an important factor.

6.4.2 Agility Management and Its Effects on Innovation

Agile methods have not only positive effects but also create several challenges. First, we will deal with the positive aspects of agile management which we identified as cooperation, customer focus and internal product and process innovation, see Fig. 6.4. Our research concludes that that with agile transformation teams cooperate better and are thus better equipped to innovate products and processes. Furthermore, interaction with the customer was also identified as an advantage, since operating agile the customer focus is in the forefront.

Fig. 6.4
figure 4

Advantages of agility on innovation. (Own illustration)

Full size image

This citation explains one interviewee’s view of customer focused innovation:

I think what we do much better today through agility is really to innovate in a customer-oriented way, hey, what does the customer want anyway, where should it go, and then to iterate again, in other words to be prepared to say, hey, the idea wasn’t so good, but that’s not bad, we’ve only invested a few thousand francs, not yet a million, so we can change it again and thereby actually have a broader portfolio for innovations, many of which will fail, but some of which can really take off because they are very close to what the customers really need. 31:53 (I12)

As mentioned previously not everything is positive in the realm of agile management. The following challenges were identified: innovation attitude & mindset, innovation focus, error & learning culture and innovation support departments (Fig. 6.5).

Fig. 6.5
figure 5

Challenges of agility on innovation. (Own illustration)

Full size image

Firstly, the results suggest that without the right innovation attitude & mindset, agile methods do not achieve their full potential. In other words, if teams are not motivated to innovate because they do not have the (time) resources to do so, agile methods have no positive effect. In addition, some employees experienced of not feeling “free enough” to innovate and are therefore not willing to leave their comfort zone. Secondly, and quite paradoxically, it was indicated that with the agile transformation the primary focus shifted to a focus on performance and delivery of output, emphasizing daily operations rather than strategic development and innovation. Thirdly, the cultural shift towards a learning culture was not made and so the tolerance of mistakes and errors remained low despite the existence of agile methods. Last but not least, it was indicated that teams lacked support, for example a department where everyone could turn to for help regarding innovation.

The following cites not only the lack of error & learning culture, but also the lacking innovation focus:

I see this rather critically at the moment, the point is that at the moment there is more pressure to deliver. Especially with Scrum, the point is the burn down rates, so Scrum clearly has the focus on delivery instead of innovation. Scrum is about working through your story points as efficiently and well as possible, it’s not about getting innovative solutions at the end of the day, but more about working solidly with your people to solve problems based on the requirements. The simpler, the better i.e. in the end the team is successful that has implemented its story points with as few mistakes as possible, with no mistakes 37:37 and I don’t think that now and with us that’s exactly what happens because the windows that are also set aside for innovation are also used to implement further stories 38:01 I don’t know how many innovations that promotes? I can imagine, the vessels to promote innovation, but then you also had to live them or demand them like the inspection adaptivity, the IP Sprint, Innovation Sprint is currently simply used to deliver more stories.38:40 (I7)

In addition, the emphasis on decentralization within an agile working environment was experienced by some as drawback for introducing radical business innovation. The following quote illustrates this contradiction between the advantage of decentralization for innovation emergence on the one hand and its disadvantage for the implementation of innovation on the other.

But at the moment it’s difficult to really drive radical innovations or bigger things because it’s like being caught between two stools and sometimes it gets lost in the day-to-day business 22:15 and we also have a very hard time with innovation because… there are simply different framework factors that play into it, so where do we want to innovate? At the moment, a lot happens decentrally, which is great, but to really drive the big innovations or to have to think from the customer’s point of view, then perhaps a central coordination or the definition of innovation would be important and really put the focus on that 22: 47 That’s a bit of a danger that I see now with the agile transformation and the very strong decentralisation of innovation, that small incremental issues are tackled everywhere, that topics are triggered by innovation, but you have to be careful not to lose sight of the big picture, that you really drive innovation across the board and then set the focus so that you don’t get bogged down and also have the necessary impact 23:13 (I10)

To summarize the use of agile methods has an impact on the innovation. However, and quite paradoxically, rather than just supporting innovation, it can lead to the contradictory effect, making it harder for the teams to innovate. In the final next section of our findings, we zoom in on some of those contradictions that the teams experienced.

6.4.3 Experienced Tensions Between Agility Management and Innovation

Of particular interest were the practical tensions and contradictions that respondents experienced in their everyday working context between the demands of agility on the one hand and its effects on innovation on the other. To fully realize the promise of agility for innovation and avoid the potential pitfalls it is important to better understand the underlying dynamics. Figure 6.6 provides a summary of the main contradictions that are labeled “group innovation”; “group decision-making” and “group quality”.

Fig. 6.6
figure 6

Contradictions between agility management and innovation. (Own illustration)

Full size image

As shown in Fig. 6.6, the first and most important set of contradictions is labelled “group innovation”. It includes: “Outcome versus Innovation”; “Outcome Conditions versus Being Innovative”; “Delivering versus Best Innovation Solution” and “Daily Business versus Innovation”. The defining dilemma across this first set of contractions is the strong focus on delivering outcomes and managing daily business. Dealing with everyday operative issues often requires different priorities than focusing on agility and innovation so that daily tasks get in the way of innovation. It suggests that the introduction of agility methods does not automatically increase the innovation capability of firms. Rather, agility is a long journey that requires a comprehensive transformation and new approach across all organizational levels. For this reason, highlighting these contradictions provides important insight for any organization which is in the process of an agile transformation.

The following citations represent the first group of contradictions <<group innovation>>:

We have a lot, the management says you should be innovative but still the products have to be delivered at the same time interval or that you should work agile and respond to the customer but when it comes to adapting something you suddenly have a “summer freeze” or a “winter freeze” and you can’t roll out anything … or for several weeks, we had no authority to develop something new and get this into production, I think that is a bit contradictory 1:00:15 (I3)

So I think the point with Scrum is that Scrum is very much focused on delivering story points, features 1:00:15 From my point of view, part of Scrum works great in IT, but innovation or things like that can be prevented from prioritising delivery …. 1:00:33 From my point of view, Scrum has a very strong effect that you deliver. With innovations, you need more time or you can only implement 15 story points instead of 20, but you might be 4 steps further along. (I7)

The second group <<decision-making>> suggests that despite agility empowering decision-making, teams continue, in practice, to be limited in their decision-making power when it comes to innovation. In fact, the findings suggest that decisions regarding innovations continue to be taken by management. In other words, while the organization under study has formally introduced agile methods, those methods are not fully translated “into practice” as old hierarchical structures continue to be in place, leading to tensions between the expectation of empowerment and the reality of still having to get clearance from management. Below are two examples that illustrate these tensions from the respondents’ point of view:

You still have no matter if somehow in one area of the company, agile unit exists, it’s still, it’s the top management, if the top management says, you yeah… don’t do it, then we don’t do it 1:10:57 and that’s not agile then. (I9)

Yes, top down vs. bottom up, I think it’s one of these contradictions that you encounter, that you want to give more competences to the employees, but sometimes you can’t because of organisational circumstances and then you have to decide top down again, so to say 47:47 concrete example, we had a reorganisation, new team formation and we wanted to make it democratic i.e. employees choose. However, there were organisational circumstances that led to the top down having to override again and that’s a bit, you have to find the balance, I think 48:12 it’s not easy. (I14)

The third group of contradictions labelled “quality” is closely related to the first group “innovation”, as innovation requires delivering new competitive high-quality products or solutions on the market. It should be noted that delivering high-quality products as well as being innovative within an agile working environment such as the Scaled Agile Framework (SAFe) was identified as hard or almost impossible.

Since the respondents emphasized that SAFe’s strong focus on features, quick delivery of products combined with the existing challenges of daily business tend to lead to a work overload. The lack of space and time for innovation was seen, again quite paradoxically, as a direct consequence of operating within the Scaled Agile Framework (SAFe). The following citations provide a good illustration of that tension:

Many, there are many. Productivity vs. quality, are you quick to deliver a lot of the product and neglect the quality or do you want to keep the quality high and deliver a bit less, these are two tensions we are dealing with 1:14:18 (I13)

Yes, there is always this opposite pole, a bit of First Time Right and Agile fast fast … bringing something 80% to the market, and for our company it is a challenge, because our brand stands for quality 48:55 (I14)

The pressure to perform and deliver limits the capacity and resources to drive innovation with potential long-term negative effects for the organization. It can thus be concluded that the implementation of agile methods needs to be aligned with an explicit focus on innovation to avoid latent tensions.

6.5 Discussion

Our study provides an in-depth understanding of the everyday practices and dynamics around agility and innovation in an IT firm. Rather than promising only positive outcomes, the findings provide a nuanced picture illustrating the key factors and challenges that companies face when implementing agile methods when attempting to foster their innovation. As we show, to be innovative within an agile working environment is not easy, because the new set-up can emphasize the delivery and continuous improvement of existing products and services rather than to innovate new ones. Furthermore, working with the agile methods, teams sometimes feel overloaded and do not have the time and resources to drive innovation. For example, scrum as an agile method includes the last sprint that is meant for innovation, but in the reality, almost no team uses this sprint for innovation, but rather it tends to be employed for last minute adjustments to the product. This leads to the paradoxically situation that the introduction of agility can decrease rather than increase innovation.

Another aspect that we deem highly relevant is the importance of attitude and mindset. An insufficient innovation attitude and mindset were highlighted in different parts of this research as one of the most important elements for encouraging innovation. This underscores the need to link the introduction of agile methods to a broader cultural change that supports proactive behavior and self-efficacy. Unfortunately, in the case study we explored, this element of the agile transformation did not receive sufficient attention. Furthermore, the need for an innovation support department is questionable since innovation should be decentralized within an agile organization. Moreover, the lack of management communication and top-down transparency were also considered barriers to innovation. Many decisions related to innovation continue to be made by management, whereas agile transformation should shift decision making power to the teams. Agile working environments should feature a learning culture. However, due to the history of the organization as centrally organized firm that did not focus on developing a learning culture this transformation was not fully put in place.

In contrast to the existing research, agile methods do not automatically lead to more innovation but rather serve as one among several support functions that drive innovation, rather being the single most important factor. In fact, highlighting the tensions, we noted how the implementation of agile methods without transforming the culture can lead to tensions and detrimental effects. We also found that employees, their mindsets and attitudes towards innovation as well as the available resources (time, space) are of greater or at least equal importance than agile methods.

Overall, we can conclude that if an organization chooses to implement agile methods it should do so consistently. If it fails to do so, the implementation of agile methods could inhibit the process of innovation.

Based on our findings and a comparison with existing literature, we identify areas for future improvement that are likely to support innovation in an agile working environment. We outline those suggestions below.

6.5.1 Practical Implications: Areas of Improvement

In line with research by Lawson and Samson (2001, p. 388) we identified areas for future improvement that may help the case organization achieve heightened agility and improved innovation processes (for overview see Fig. 6.7).

Fig. 6.7
figure 7

Categories of innovation. (Own illustration according to Lawson and Samson (2001, p. 388))

Full size image

The results suggest that the success of agile transformation is closely linked to “organizational structures and systems” and is the priority for the suggested improvement. Following this, management of technology was identified as another important area, because technology is the DNA of the company in this investigation. Apart from this, three additional areas were identified that would help to encourage innovation:

  • creativity & idea management;

  • vision & strategy & organizational intelligence;

  • and culture & climate.

Although some decentralized activities already address creativity & idea management, several respondents pointed out that this needs improvement. Creativity & idea management is an important category and can motivate and inspire employees to come with new ideas.

Vision & Strategy and Organizational Intelligence is another important area for improvement. According to the analysis of the results, the teams within an agile working environment are lacking a vision that provides them with a sense of direction for promoting innovation and developing their innovation capability. Not only vision & strategy, but also the future focus on the organizational intelligence and the use of its full potential was indicated as a “must have” for the organization. The focus on the organizational intelligence will help companies develop learning and smart organizations where innovation is the focus and obtaining resilience through innovation the main goal.

Culture & climate is the final area for future improvement. In line with the analysis of the results we found that teams need a good work climate, where they feel secure and free in a “culture of trust” indicated as an essential for creating new competitive advantages.

In conclusion, our study provides insights in the contradictory reality of agility and innovation and gives helpful guidance for organizations that are in the process towards an agile transformation. Taking our findings into account and focusing on the identified areas for improvement will help organizations promote agility and innovation in order to stay competitive in the coming industry 5.0, in in which agile innovation processes are likely to prove even more critical for resilience.