The three main leadership challenges found in our data is summarized below. The challenges are shown below together with quotes and a discussion to support the claims.
3.1 Team Maturity
When the interviewees compared less and more mature teams, the latter were said to tailor their own agile process based on contextual knowledge. One interviewee saw a strong connection between more mature teams that have met for a longer period of time and how much initiative and responsibility they take for the process and collaboration, implying that their leadership role is easier and more consultative.
“They reinforce the practices within the teams themselves.” [Project manager and initiator of the agile approach]
Another indication that the level of team maturity is highly influencing the appointed leader’s leadership style in agile teams, is the fact that more mature teams were said completely adopt the agile practices they find useful, while less mature teams need reinforcement of the practices, otherwise they are reluctant to use them. To enforce the practices is then something the leaders describe that they must do for the team.
“Those are very visible high performing teams self-directive aspects vs. those needing reinforcement of the practices for them to be there.” [Project manager and initiator of the agile approach]
A key seems to be to suggest best practices by the leaders for the team, have a minimum of what is allowed, and then let the teams tailor their process themselves. In such a way, less mature teams will resort to safety by adopting a predefined process but can then redefine their process as the team matures.
“We really try to get the teams to focus on staying within the framework, but they have latitude and liberty within that framework to, based on their own team style or team makeup.” [Project Management Leader]
Another interviewee clearly stated that even the most self-organized teams were different initially. The “agility” simply had to wait for the team to mature, which means that the needed leadership style is different across time.
“Yeah, these days they don’t need me in order to work. These days I am really a facilitator and the team is absolutely able to the normal Scrum process without me. They don’t need any guidance any longer so I can easily go on vacations for 2–3 weeks, that’s not a problem. /.../ Of course in the beginning I had to stop them in the dailies and say ‘Stop discussing solutions,’ just the tasks, please, and the three questions bla bla bla. Now it’s more or less routine.” [Scrum Master]
This indicates that the self-organization of teams emerges over time along with the team maturity from a psychological perspective, which also implies that the leaders must take on the function of leadership initially before that function can be shared. In addition, the built in flexibility of the agile processes is also something teams need maturity in in order to leverage in the intended way. One interviewee stated that more mature teams can easily change their process if asked, which is not something the less mature teams could do in the same way.
“We are trying to adopt Kanban. But there are other teams here that have only worked with Kanban for a few months. They tried it because they saw some problems we had with Scrum. Some of the teams matured faster (mainly because they didn’t break up the teams all the time as we do here) and they changed to Kanban.” [Scrum Master]
For teams that are mature, and where the members are dedicated and have set clear roles, the interviewees saw that they could adopt self-organization and team agility without many issues, putting less focus and dependence on them as the designated leader or manager.
“The team we have is an extraordinary team with a very open mindset and a very innovative team and always open to new things, so they were very open-minded so that was not much to say, and they trusted me.” [Project Manager Lead]
To summarize, the first found challenge is that the agile leader needs to take a step back from mature teams and instead facilitate the team’s work process in relation to the surrounding ecosystems of the teams. However, in order to implement agility in the less mature or newly formed teams, they also need to provide a lot more direction and guidance in order for such teams to become self-organizing agile teams.
3.2 Team Design
The team design process is very much connected to team maturity as described in the previous section. However, the following quotes are in this category because they are a symptom of a lack of team design from an external role, not a symptom of the team being new.
When teams were not given an initial structure, some interviewees were surprised and frustrated when newly formed teams expressed a need for clear and directive leadership.
“It was the first retrospective that we had, they say that they lack some leadership there and then during the retrospective we were talking like ‘no it’s not leadership that you need, how about that you decide how you will do things and the new habits that we will create in the next sprint.’ What kind of agreements that we have to have for solving that problem, and we’re saying that the lack of leadership is the result. That’s one thing that appears a lot because we are growing and new teams are being formed.” [Founder of an agile company]
New team were also described as being open to any work practices and lack the insight into what is useful or needed in their context. This entails that the leader needs to step in and guide the team in making such decision, something that was described frustrating for the leaders since they did not expect that in the agile approach. However, the team must get help in its initial design since no team members can know the context simply because it is new to them.
“Yeah, for teams that are younger, like formed more recently, they tend to be more open to all the practices, but they don’t have the experience to decide which practices that would be the best.” [Founder of an agile company]
Also in relation to new teams, one interviewee changed the agile practice of volunteering for task because the team was not ready to take on that responsibility on a team level. This implies that the leader felt the need to step in and be more directive, however, with the expressed frustration that such a leader behavior is not appropriate in the agile context. Teams, though, seem to need help in designing work processes initially.
“In agile people should volunteer for tasks, but in most cases here we are obliged to... we do task assignment, by me or the person who already worked on this item takes this item. I know that this isn’t a good practice in agile, but we do it for our team and for more productivity, but also for responsibility.” [Scrum Master]
An interviewee from hardware development also highlighted that, since the company is expanding all the time and therefore consistently gets more team members, it is difficult to design and build self-organizing teams, which is a core part of the agile approach and frustrating for them as leaders.
“You have to be very involved to be able to get a clear direction from it, and that’s hard, especially because our group has grown so much and we get new people all the time who don’t have that direction from the beginning. /.../ The further the team gets, and the more you have worked, the more autonomous the group becomes.” [Scrum Master and Software Developer]
We also found support for the distinction between what is an organizational and enforced structure and what is up to the teams themselves, i.e. the right balance in team design. Teams with no provided structure were described as much less effective but it is about providing the right balance of flexibility and control by the people in leadership positions.
“The teams can change as they want [in the process] all the time.” [Supporting role]
Traditionally, it seems like most software development processes did not have any team reflection sessions by default. The agile process often adds the retrospective meeting, which is a structure for team reflection, which was described as helpful for the leaders in order to improve the teamwork.
“The developers, I think, also feel that it provides them with a preset structure within which they can communicate with each other; they don’t have to set up a meeting to do this. We have our Scrum identify that they need to meet to talk about something, and then they do. So it puts things in place for them and they don’t have to think about it.” [Project Management Leader]
To summarize, the leaders need to help transitioning teams to design their new agile structure and ways of working. If teams are not ready to tailor their own process, the leaders provide suggestions and best practices, but keep stating that the team should continuously improve their process based on what they learn about their ecosystem. This means that, when the teams are set up, they need an initial predefined team design and then be given the possibility to tweak their processes when the team is ready.
3.3 Culture and Mindset
The third and final challenge was that the leadership also needs to be adjusted to the existing more traditional structure and culture of the company. We found that even if members and teams can adopt self-organization, the context might hamper this way of working.
“If the whole company was agile in the end, the teams could be more independent and talk to the business area more themselves, but now I spend a lot of time trying to make decisions on what the teams should do.” [Scrum Master]
All the interviewees spoke about the different agile roles changing dynamically based on what is really needed at a point in time. A project manager acts as a Scrum Master, or trying to help people to not fall into old patterns of behavior in relation to the old structure instead of the new agile process.
“After the first project I was able to spend time educating them at the beginning, but had to make almost daily conscious efforts at reminding them, or educating them with when I could see that their though process was tending toward waterfall. So I’ll try to point out: ‘oh actually, let’s think about it this way.’ Or, you know, helping them with definitions like what the basic function of the daily Scrum for example and remind them of that. That it’s not a status meeting, for example.” [Scrum Master and Project Manager]
One participant from the hardware-focused development also highlighted the fact that all individuals do not have the same possibilities, or motivation, to adapt to the new ways of working. The participants from hardware and software development differed in that the hardware-focused interviewees focused on the “old” ways of working as compared to the new agile way, while the software-focused participants reflected more on the team’s place in the company as a whole.
“Partially, I believe it’s due to... from what I’ve heard, that [a team member] that is a bit more senior and has had previously bad experience from the agile ways of working in other areas and therefore doesn’t think it worked well and actually only sees the negative aspects. [The team member] just does this because everybody else wants to do it, but does not think it will work. And therefore it does not fully work, because you... it’s hard. I think it’s really hard to have one leg in it, and one outside, you know...” [Scrum Master driving the agile transformation]
From the hardware-focused participants, they described their agile implementation as something that needed to be agile in itself. That means that they had to related their new process to also fit with the old, which was described as challenging.
“We have tried to see the work process as something agile and adaptable too! Now there are more clear toolboxes with how to work in an agile way in the company. When we started that didn’t exist, but, I mean that, when you talk about Scrum as an agile method it’s not anything more than a toolbox with different practices, and you have to try to pick what suits your organization and your... we started with that mindset that we would have ‘what is the smallest part of scrum we can pick?’ or ‘what is the smallest set of tools we can make use of?’ and then, I guess, with time we have realized that, yes, ‘we need this’ or ‘we need do do that one too’. [Scrum Master driving the agile transformation]
To summarize, the participants’ leadership style is also adapted to where the company is in its agile journey, not just the internal process structure, as in the previous theme. The leaders act as both more traditional managers and as more agile (i.e. contextually adapted) leaders depending on what is needed at that point in time.