Keywords

1 Introduction

When compared with classic hierarchical and Tayloristic management, agile software development is a radically different way of organization. While early agile methods like XP and Scrum aimed at the team level only and more or less ignored the organizational context, nowadays whole organizations “go agile”. Such a transformation requires taking into account more than just core teams: management processes and responsibilities, the underlying organizational culture, and leadership will be affected, the more an organization implements agile software development [1,2,3].

Early approaches to agile software development did not explicitly address leadership. In fact, leadership or the leader’s role are not even mentioned in the original Agile Manifesto and its twelve principles [4], or in the latest version of the official “Scrum Guide” [5]. On the other hand, self-organization and autonomy are at the core of agile teams. It is striking that these approaches seem to ignore the wider organizational context, and especially the role and responsibilities of “classic” hierarchical leaders or line managers. While the classical leadership roles might have changed, the tasks of leadership have not disappeared. But how are they executed in agile teams and organizations? How are they adapted in order for agile methods to work in an organizational context?

Recently, industry has become more aware of this new challenge. The “Agile 2” movement postulates that the “largest defect in agile thinking regards the role of leadership” [6]. They propose a new set of values and principles, many of which directly concern leadership and its role in agile organizations. In the Harvard Business Review article “The Agile C-Suite”, the authors state the need for a new leadership approach [7]. Such practitioner-led endeavors manifest the change in the leadership role and maybe the need for a better understanding of it. On the academic side, while some studies have investigated questions around “agile leadership”, the overall body of research is still rather thin [9]. In this paper, we present our findings from an online survey about agile software development and leadership in IT companies. We show how leadership styles and practices change in more agile contexts. We address the following research questions:

Q1: Do organizations implementing agile software development show less hierarchical leadership and more shared leadership than less-agile contexts?

Q2: How does transactional and transformational leadership differ in agile vs. less-agile software development?

Our results show that while there are, broadly speaking, shifts from hierarchical to shared leadership and from transactional to transformational leadership, reality seems to be more complex.

The rest of the paper is structured as follows: In the next section, we present our theoretical framework. Section 3 explains our research design and measurement of constructs. In Sect. 4 we present the results of our study, followed by a thorough discussion and final conclusions.

2 Related Work

Leadership is a mature area of organizational research underpinned by numerous theories and approaches [8]. However, in the agile software practice literature, leadership is rarely addressed explicitly. Guidelines such as the Scrum Guide [5] only briefly mention servant-leadership and self-managing teams. In academic literature, a few studies have been conducted on the role of leaders and leadership in agile software development. A recent systematic literature review [9] categorizes studies into three groups: a) studies based on leadership theories, b) tangential theories and models where leadership is included, and c) leadership styles. Leadership theories used include full range leadership theory (transactional, transformational, and laissez-faire leadership), a leadership taxonomy, complexity leadership theory, and role theory. Leadership styles explored include adaptive, shared, transformational, ad-hoc, mentor, servant, situational, expert, and super leadership. They conclude that while research on agile leadership has grown since 2005, it is still a nascent research area in which more empirical research studies are needed. They did not find a common view, but indicate that the focus moves away from hierarchical and bureaucratic leadership, and that leadership needs to change as agile teams change and mature. Yang et al. [10] asked traditional and agile project managers whether a transformational, transactional, or laissez-faire leadership approach best suited their projects. They found more need for transformational leadership in agile projects than in traditional ones. A paper by Gren and Ralph [11] reports on a small qualitative study with self-described leaders in agile development projects, finding that leadership is shared with teams, builds a sense of common purpose, and adapts to organization culture. Spiegler et al. [12] undertake a grounded theory study of Scrum Master leadership and identify nine roles that are transferred from the Scrum Master to the development team as it matures.

For this paper, we focused on two dimensions of leadership, namely leadership style (transactional or transformational) and leadership locus (hierarchical or shared) as they are well-researched, classical concepts that encapsulate some of the key differences between traditional and agile organization.

Fig. 1.
figure 1

Leadership locus/style matrix. Vertical axis is leadership style (transactional/transformational). Horizontal axis is leadership locus (hierarchical/shared)

Full size image

First, a long-established body of leadership theory pertains to the style with which leadership is executed. Classic concepts distinguish transactional and transformational leadership styles [13]. Transactional leadership is, in essence, the idea of leading people by designing and adjusting an economic contract between leader and follower. Labor and its output are traded for a salary or for opportunities for promotion. The function of transactional leadership is to set, monitor and adjust goals, expectations and incentives. In contrast, transformational leadership describes a relational contract rather than an economic one. Avolio et al. [14] define transformational leadership as “leader behaviors that transform and inspire followers to perform beyond expectations while transcending self-interest for the good of the organization.” The function of transformational leadership is therefore to create a sense of mission and purpose within those being led.

Second, it has long been recognized that leadership is not just situated in an individual with formal authority, but can rather manifest in different loci like context, team, dyads, etc. [15]. In our paper, we focus on the leader (individual with formal authority) and on the team (group of people interacting with little or no regard to formal hierarchy) and call these loci hierarchical leadership and shared leadership, respectively. Shared leadership has been defined as “a dynamic, interactive influence process among individuals in groups for which the objective is to lead one another to the achievement of group or organizational goals or both” [16]. In contrast, we define hierarchical leadership as influence processes occurring in a relationship characterized by formal authority (e.g., a line manager and their respective employee). Leadership can thus be found in at least two places, or loci: in the hierarchical relationship between formal leader and follower, and shared (distributed) among team members. This structure of two leadership loci and two leadership styles is illustrated in Fig. 1, with locus on the horizontal axis, and style on the vertical axis.

It should be noted that both transactional and transformational leadership were originally thought of as personal styles, existing purely on the individual level of the formal leader. Following Schein [17] we argue, however, that both these leadership styles can also be seen as important leadership functions in the organization, which can be served by different loci. The goal-setting and -adjusting of transactional leadership can therefore (theoretically) also be accomplished on a team level, as can the inspiration, creation and affirmation of a sense of mission typically attributed to transformational leadership. Using these two distinctions ‒ hierarchical vs. shared leadership and transactional vs. transformational leadership ‒ we can now theorize and derive questions about changes in leadership in less agile vs. more agile contexts of software development.

Reading many agile concepts and methods could lead one to assume that only transformational and shared leadership is important in agile software development. Most agile methods still presume the existence of a formal leader (sometimes called “line manager”), but their importance is reduced and many leadership tasks are distributed among the development team, using specified roles, as well as principles of self-organization. Because of this, and because of a presumed general occurrence of agile methods in “flatter” organizations, one would assume that agile software development is correlated with shared leadership. But does this also mean that hierarchical leadership decreases or do both exist simultaneously? Regarding leadership style, does the importance of short-term-iterated planning and adjusting of goals, inherent in agile principles, relate to a decrease or increase of transactional leadership? Does the relevance of transformational leadership increase in more agile software development, because creating and maintaining a sense of purpose becomes more important in self-managed organizations, as some have argued [18]?

We found that using our theoretical lense of leadership style and locus produced a number of interesting issues, all worthwhile pursuing, which led us to apply a more explorative approach. We do not aim to provide definitive answers to any of these questions, but rather want to open up avenues for further debate and research. We therefore decided against testing specific and focused hypotheses and formulated the following research questions instead:

Q1: Do organizations implementing agile software development show less hierarchical leadership and more shared leadership than less-agile contexts?

Q2: How does transactional and transformational leadership differ in agile vs. less-agile software development?

3 Research Methods

3.1 Data Collection and Sample

This study is based on the online survey “International Agile Study 2018/2019” conducted in Switzerland, the United Kingdom, and New Zealand in 2018 and 2019 regarding the usage of development methods and practices in the IT industry, and about the impacts of applying agile methods. For a detailed description of the survey instrument see Kropp et al. [19]. The survey addressed both agile and plan-driven companies, as well as both agile and plan-driven IT professionals, or any hybrids. There were in fact two independent surveys: one for companies, and one for individual IT professionals. In the company survey we targeted representatives of the company or the development department of a company, i.e., typically upper management level. The addresses of the companies were collated from participating IT associations from all involved countries as well as from our own institutional databases. To ensure a company was represented only once in the company survey, we sent personalized links to one management representative of each company. The IT professional survey was anonymous, and we invited wider participation. We sent invitations with a link to the survey via email and through professional social media like LinkedIn and XING (a career-oriented social networking site popular in German-speaking markets). Participants were typically directly involved in software development, and we describe the demographics in the section below. The survey was a general survey about the state of agile software development, either in IT companies or in companies with significant IT activities (e.g., banks, insurance, chemistry). The questions covered a broad range of aspects in agile software development and were the same for both surveysFootnote 1. In this paper we focus on the analysis of the leadership questions.

3.2 Participants

The survey was answered by 199 professionals and by 88 company representatives. Since we wanted to study shared leadership, we removed high-level leaders (because they most likely are not part of a real team), and we excluded all those who did not answer any of the leadership questions (missings). The final sample was N = 200 (20.5% of which from the company and 79.5% from the professionals’ survey). The average age of the participants was 42 years with an average IT experience of 18 years. The participants were IT professionals working in various sectors like retail, medical and health, finance, transportations and shipping. Of the 200, 75% were male, 12% female, 3% explicitly preferred not to say and 10% did not indicate gender. The participants mainly came from the organizing countries, but we also had answers from Austria, Germany, and the United States.

Table 1 shows the roles of the participants in their company.

Table 1. Roles of participants.
Full size table

3.3 Questions, Constructs and Analysis

Extent of Organizational Agility.

In order to measure the extent of agility of an organization, we used the single-item question: “Is your organization currently practicing plan-driven or agile software development?” with a 5-point Likert-scale with the following anchors: (1) all plan-driven, (2) mostly plan-driven, (3) both plan-driven and agile, (4), mostly agile and (5) all agile. Note that the question specifically referred to software development rather than other aspects of the organization. To gain further insight, we also asked which agile methods were used, if any, the number of years of the organization’s experience with agile methods, and to what extent participants were satisfied with the organization’s methodology.

Leadership Loci: Hierarchical vs. Shared Leadership.

In order to measure hierarchical leadership, we used the questionnaire from Ismail et al. on transactional and transformational leadership styles [20], which is an adaptation of Bass and Avolio’s Multi-Factor Leadership questionnaire [21]. To assess shared leadership, we re-formulated the items by replacing “my direct supervisor” with “my team.” This means that each participant saw 20 leadership questions, 10 for hierarchical locus and 10 for shared locus, each with 5 for transactional style and 5 for transformational style, as shown in Table 2. Each question was answered using a 5-point Likert scale from 1 (Strongly Disagree) to 5 (Strongly Agree). The responses were combined, resulting in an aggregate score from 1 to 5. The internal consistency of the answers was good to very good: for the four combinations of locus and style, we report Cronbach’s Alpha in Table 3.

Table 2. List of items used to measure leadership: answered using Likert scale from 1 (strongly disagree) to 5 (strongly agree).
Full size table
Table 3. Four sets of responses for locus and style, with Cronbach’s Alpha showing good internal consistency.
Full size table

Analysis.

Our approach in this study emphasizes understanding and is principally exploratory. While we do address our research questions, we therefore refrained from proposing and testing specific hypotheses. Our analysis consists mainly of inspecting descriptive results, correlations, and graphical comparisons of distributions. We hope this approach serves to inform future work that is then able to frame and test hypotheses.

4 Results

The participants worked in companies which are experienced in agile software development, with a large majority practicing Scrum alone or in combination with other methodologies. Most companies (74.8%) have been practicing agile software development for at least three years. The vast majority of the participants (81%) worked in organizations which are at least slightly experienced in agile software development, with 28% very experienced, 31% moderately experienced, 28.5% slightly experienced. Only 5% stated that the company had no experience with agile software development (7% did not rate the experience of the company).

The extent of agility in software development varied across the organizations: 13 participants (6.5%) reported all plan-driven software development, 25 participants (12.5%) mostly agile, 78 participants (39%) work in organizations where they practice both plan-driven and agile software development, 65 (32.5%) participants report mostly agile, and 19 (9.5%) participants report all agile software development. Elsewhere in our survey we asked questions about use of a range of agile practices, and we found strong correlations between that data and the level of agility reported.

The companies used a broad range of agile methodologies (Scrum, XP, SAFe). Most companies claim to follow the Scrum methodology (47%), followed by Kanban (8.5%), combined Scrum and eXtreme Programming (6.5%) and DSDM/AgilePM (6.0%). 12.5% used the free text option and most of them stated that they use a mix of different methodologies; 0.5% did not state the methodology of the company. The majority (59%) of the participants were satisfied with the company's current methodology. Only 11.5% of the participants were unsatisfied about their company’s current methodology.

In Table 4, we display descriptive statistics for the extent of agility and leadership by locus and style. On the right of the table, we display the correlation between extent of agility and leadership, showing Spearman’s rho and the p value (uncorrected for multiple tests). Although the intent of our study is principally exploratory, rather than hypothesis testing, we report p values as an indication of the rarity of the results in order to inform future work.

We can see some general differences in the data for both leadership loci and styles. In every case where we distinguish loci, shared leadership is consistently rated higher than hierarchical leadership. In every case where we distinguish styles, transformational leadership is rated higher than transactional leadership. For the four specific cases (last four rows), ANOVA and Tukey HSD tests show all differences to be significant.

Table 4. Descriptive statistics for Extent of Agility, for leadership by locus and style, and correlation between Extent of Agility and leadership (for measures combining loci or styles, we only include cases where we had responses for each).
Full size table

Examining the relationship between the extent of agility and leadership, we can see that, in general, over both loci and both styles, leadership is related to the extent of agility (rho = 0.277, p < .001). At a finer level, however, we can discern several differences. The strongest relationships are with a shared locus (overall rho = 0.370, p < .001) and with transformational style (overall rho = 0.321, p < .001). The hierarchical locus does not show a correlation overall (rho = 0.111, p = 0.117), and in particular no correlation is seen for a hierarchical locus and a transactional style (rho = 0.008, p = 0.914). To examine the patterns, we created the series of graphs shown in Fig. 2. Each of the four graphs corresponds to one of the four combinations of locus and style, arranged as described earlier in Fig. 1. Each graph shows five boxplots, one for each of the extents of agility (All Plan-Driven to All Agile), showing the rating for the leadership locus and style specified.

Fig. 2.
figure 2

Plots showing relationships for each of the four pairings of locus (hierarchical and shared) and style (transactional and transformational). The boxplots show the relationship between the Extent of Agility (horizontal axis), and level of Leadership (vertical axis). [Each boxplot shows the median (dark horizontal line, the inner quartiles (colored box), the outer quartiles (whiskers) and outliers (circles).]

Full size image

The pattern for hierarchical locus & transactional style (bottom left) shows an initial rise from all plan-driven, but then a fall for mostly and all agile, corresponding to the lack of correlation (rho = 0.008, p = 0.914). However, it may be important to note that while there is no correlation: the measure is fairly consistent, and even for all agile, hierarchical-transactional leadership is rated midway on the scale. Hierarchical locus with transformational style (top left) shows a modest rise (rho = 0.179, p = 0.012). Shared locus with transformational style (top right) shows a consistent and strong rise (rho = 0.370, p < 0.001). Shared locus and transactional style, interestingly, also shows a strong and consistent rise (rho = 0.311, p < 0.001).

5 Discussion

5.1 Interpretations of Our Findings

We set out to study the relationship between leadership style and locus and the extent of agility in agile software development, and we found strong correlations between some aspects of leadership, but not all of them.

Our first research question concerned hierarchical and shared leadership and their connection to agility. Our data show that while shared leadership is (somewhat unsurprisingly) strongly related to more agile contexts, scoring very high in all-agile software development, the results are a bit more nuanced regarding hierarchical leadership. Overall, the intensity with which people experience hierarchical leadership does not change much as software development becomes more agile. Differentiating between the transactional and transformational style within the hierarchical leadership locus showed us that transformational hierarchical leadership increases slightly, showing a weak correlation, whereas the relationship between the transactional leadership style and agility resembles an inverted U-shaped curve. In essence, it is fair to say that in agile software development, hierarchical leadership is still present – especially in combination with the transformational style. Our data do not tell us whether this generally is positive – it could very well be that agile software development with less hierarchical leadership outperforms other practices. Nevertheless, it is still surprising to see that hierarchical leadership does not wane much.

With our second research question, we looked specifically at changes in leadership style as software development becomes more agile. We found distinct evidence that transformational leadership is related to the extent of agility in software development. This effect is very strong for shared transformational leadership and weak (but still present) for hierarchical transformational leadership. We also found that shared transactional leadership markedly increases in more agile contexts, while for hierarchical transactional leadership the above-mentioned inverted U-shaped relationship applies.

In our view, the two most interesting results of our study are:

  1. (1)

    Hierarchical leadership does not become irrelevant in agile software development. People experience both transactional and transformational hierarchical leadership quite strongly, even in mostly or all-agile contexts. While leadership does become more distributed, leadership executed by direct supervisors and/or line managers still holds relevance.

  2. (2)

    Transactional leadership does not become irrelevant in agile software development, either. Goal-setting, accountability and other more “directive” aspects of leadership are still very present in agile contexts, but their locus seems to shift from the line manager to being shared in the team.

As we described earlier, our questions on leadership were based on Ismail et al.’s questionnaire [20], with five each for transformational and transactional styles, and we adapted these to distinguish a hierarchical and a shared locus. To further investigate our results post-hoc, we explored correlations between extent of agility and the responses to individual questions. In Table 5 we show these correlations. One overall pattern is that almost all the correlations for the shared locus (rightmost column) are greater than the equivalent correlations for the hierarchical locus (column to the left). The only exception involves the question about monitoring performance, where the correlation is not significant for shared, but negative for hierarchical. Also, while this is the only non-significant correlation for the shared locus, there are many for the hierarchical. Moreover, with an alpha of .001, none of the correlations are significant for the hierarchical, whereas six remain significant for shared locus. Looking at the three strongest single correlations could give us some idea of what differentiates agile from non-agile leadership the most: “Setting standards to carry out work”, “encouraging to rethink never-questioned ideas”, and “taking action before problems are chronic” within the team (shared locus) seem to be good indicators for agile leadership. Notably, two of these regard the transactional style.

Table 5. Correlations between Extent of Agility and responses to individual leadership questions, by locus and style; columns at right show Spearman’s rho, where below p = 0.05 (uncorrected for multiple tests).
Full size table

Another question that arises from this in-depth analysis is the role of performance monitoring, which notably does not increase with a shared locus and seems to become even less relevant with a hierarchical locus. At least in part, the phrasing of the question as “monitoring performance and keeping track of mistakes” might be the cause of this result, as that could have a rather negative connotation for people. However, the drastically different result for this single item still raises the question: Who monitors performance in agile software development?

Looking at our results more broadly, it is also noteworthy that, overall, people experience more, or more intense, leadership (as measured with our items) in agile software development. One could have assumed that overall leadership is equally “strong” in plan-driven contexts, just more hierarchical and/or more transactional. This would have shown as a sort of x-shaped relationship in our data (as one aspect of leadership goes down, another one goes up). Instead, it seems that leadership in general is more prevalent in agile than in plan-driven software development (with the exception of hierarchical-transactional). The positive interpretation of such a finding might be that agile software development allows more people to participate in leadership processes as part of an empowerment or even emancipation process. On the other hand, one could argue that “more leadership” is not without cost, as it also means more complexity in decision-making and navigating relationships. Handling such increased complexity requires more psychological and social resources from people.

5.2 Implications for Research

The qualitative study of Gren and Ralph [11] found that self-described agile leaders emphasized the importance of shared leadership and fostering a sense of common purpose. Our results are consistent with those findings. Yang et al. [9] found that transformational leadership was more highly rated by agile managers than by traditional managers whereas transactional leadership was equally rated. We also find that transformational leadership becomes more important as organizations become more agile, but additionally that shared transactional leadership is important, and that hierarchical leadership still appears to play a part. Another consideration is the role of individual people. Gren and Ralph’s participants all claimed to be leaders, and some of their job titles appeared to possibly suggest some hierarchical authority. The interplay between a hierarchical and a shared locus of leadership for agile development may be complex and subtle.

The nature of the transactional style within agile development also needs further study [9, 10]. The issue of hierarchical-transactional leadership relates to the role of a hierarchical locus within Agile, and while this is seldom acknowledged in articulation of agile processes, it is still commonplace in practice. Another issue relates to shared-transactional leadership. Our results suggest this is stronger in mostly or all-agile teams. This might relate to some well known practices, such as XP’s “planning game” or “planning poker”, where the whole team is involved in planning, and then commits to that plan. However, especially in an organizational context, this raises issues of stress and overwork, and overall responsibility. Even in a positive context, the effects of social pressure can be serious.

In future work, it would be interesting to look at different results based on individual roles. For example, do Scrum Masters perceive shared leadership in the agile software development teams differently than developers or product owners? Such detailed analyses could reveal insights about the distribution of leadership responsibilities and its effects on software development.

In summary, we suggest there is a need for further research into the role of transactional and hierarchical leadership in agile software development. While this study has identified their continued use, without contextual research that seeks to uncover the potentially complex stories underlying their use, we can only speculate about their role and relevance.

5.3 Implications for Practitioners

Members of agile software development teams could, firstly, use our results to clear out some myths that might exist around agile leadership: that hierarchical leadership is no longer present, or that encouragement, emotional support and other ideas around transformational leadership are the only important aspects in leading an agile team. We can show quite clearly that direct supervisors still play an important role and that transactional leadership on the team level is even more relevant in agile software development. This leads to our second implication, namely that teams should understand and take to heart the nature of shared-transactional leadership: Aspects such as goal-setting, making expectations clear, and taking action before problems become chronic are key for agile shared leadership. This requires actually a very disciplined work style of agile teams. Especially Scrum Masters should not only make sure there is commitment (in the emotionally invested sense), but also that all members are aware of exactly what they have committed to. This point is also noted by Spiegler et al. [12], who identify a leadership role called “disciplinizer on equal terms” for Scrum Masters which involves them helping the team to understand for themselves the importance of discipline and focus in their work.

5.4 Limitations

We need to recognize issues relating to our sample. We invited many people to participate in our survey on agile software development, but only some chose to participate, so our sample is self-selected. In our analysis we look for relationships between the extent of agility and attributes of leadership. We need to be cautious about several aspects of this issue. We determined the extent of agility on a scale from 1 to 5 by asking participants about software development in their organization. We acknowledge this is a complex issue which cannot easily be represented as a simple ordinality. The questions from which we derive our measure of leadership are based on established instruments, but there may have been different interpretations of the wording. For example, we discuss above how “monitoring performance” might be interpreted negatively. Perhaps most importantly, our analysis uses correlation. While this allows us to determine, for example, that more agility is associated with more shared leadership, we cannot assume that more agility is the cause of more shared leadership, or vice-versa. Establishing causality would require more detailed study.

6 Conclusions

Our study was to explore leadership in agile software development, in particular the style of leadership, transactional and transformational, and the locus of leadership, hierarchical or shared. We adapted an established questionnaire instrument and examined the responses from professionals actually involved in development. Our results suggest a strong relationship between the level of agility and the impact of a shared locus, including both a transformational style and also a transactional style. The extent of agility was also (more weakly) related to a hierarchical locus transformational style, but not with a transactional style.

For future work, we would like to address the limitations and probe the key findings. We especially wish to further examine how a shared locus of leadership appears to involve both transformational and transactional leadership at the same time. Furthermore, looking at outcome measures (e.g., productivity measures, satisfaction, or perceived success of agile transformation) and their relationship to the different aspects of leadership in agile software development should prove particularly valuable.