We first provide a description of the Perform programme before presenting results on our four areas of data collection: programme organization, customer involvement, software architecture, and inter-team coordination.
The Perform Programme
The Perform programme was conducted by the public department the Norwegian Public Service Pension Fund (the “Pension Fund”), which needed a new office automation system. The main arguments for initiating the programme were public reform that required new functionality in office automation, and the existing office automation system was on a platform that was to be abandoned. When the programme started, the content of the public reform was not known. This was the main reason for choosing agile development practices for the project. The public department has about 380 employees and provides 950,000 customers with several types of services. The department integrates heavily with one other public department.
Perform was initiated to enable the department to provide accurate and timely services to customers and ensure a cost-effective implementation of the reform. Because of the reform, the programme had a strict deadline. It is one of the largest IT programmes in Norway, with a final budget of about EUR 140 million. The programme started in January 2008 and lasted until March 2012. The project involved 175 people, of whom 100 were external consultants from five companies.Footnote 6 The programme used both time and material and target price contracts for subcontractors. About 800,000 person-hours were used to develop around 300 epics, with a total of about 2500 user stories. These epics were divided into 12 releases as in Fig. 1.
As an example, release R8 contained coupling of workflow in the office automation system to the archive solution, a self-service solution for new legislation, simulation of services towards external public departments, and first data warehouse reports on new data warehouse architecture. Most user stories were identified prior to the first release but were supplemented and reprioritised for every release.
The existing office automation system was client/server-based and written in C. The new system is a service-oriented system written in Java and provides a richer interface using Flex.Footnote 7 The database from the old system was kept, but the data model was changed. The regulations and legislations are implemented in the new system as rules using JRules. The system is integrated with a new document archive and with systems from another public department. Figure 2 gives an overall description of the existing solution and the new solution.
The programme was managed by a programme director who mainly focused on external relations, a programme manager focusing on the operations, and a controller and four project managers responsible for the business, construction, architecture, and test projects (see Fig. 3):
Business - Responsible for analysis of needs through defining and prioritizing epics and user stories in a product backlog. This project was manned with product owners and a total of 30 employeesFootnote 8 from the line organization in the department. Functional and technical architects from development teams also contributed to this project.
Architecture - Responsible for defining the overall architecture in the programme and for detailing user stories in the solution description phase. Consisted of a lead architect and technical architects from the feature teams. Suppliers Accenture and Steria participated on a time & material basis.
Development - Divided into three subprojects: one led by the Pension Fund (6 teams) with their own people and people from five consulting companies, and the two other subprojects led by Accenture (3 teams) and Steria (3 teams). The feature teams worked according to Scrum with three-week iterations, delivering on a common demonstration day. The feature teams had roles such as Scrum master, as listed in Table 3. In addition to the 12 feature teams, the project had an environment team responsible for development and test environments.
Test - Responsible for testing procedures and approving deliverables from the development teams. Consisted of a lead tester and test resources from development teams.
There were also projects for communication and adoption to prepare users for the new systems, in total six projects.
As shown in Fig. 3, the programme used a matrix structure where the business and development projects took part in the architecture and test projects. This matrix structure meant that a feature team would then mainly take part in project development, while also devoting resources to projects architecture (through the technical architect), business (through the functional architect), and test (through the test responsible) as shown in Table 3.
The programme started working at the main office of the public department, but in 2009, it was moved to a separate office building located in the same city. Here, all teams were organized around tables as shown in Fig. 4.
Initially, the development process included the four phases described in Fig. 5:
Analysis of needs - This phase started with a walkthrough of the target functionality of a release and identification of high-level user stories. Product owners prioritized the product backlog.
Solution description - The user stories were assigned to epics, and the user stories were described in more detail, including design and architectural choices. User stories were estimated and assigned to a feature team.
Construction - Development and delivery of functionally tested solutions from the product backlog. Five to seven iterations per release.
Approval - A formal functional and non-functional test to verify that the whole release worked according to expectations. This included internal and external interfaces as well as interplay between systems.
To keep the schedule of the project, solution descriptions needed to be ready in time for the feature teams. This meant that releases were constantly being planned, constructed, and tested. Thus, given the roles in Table 3, a feature team would constantly be engaged in construction for release “n”, approving delivered functionality in release “n-1”, and analysing needs for the next release (“n + 1), as shown in Fig. 5. After approval from the programme, new releases were acceptance tested, set in production, and underwent an approval phase before being accepted by the operational IT section of the department.
The main arena for customer involvement in Perform was the interface between business and architecture subprojects and the analysis of needs and solution description phases. We found three key characteristics of this work: first, that the solution description was developed using teamwork; second, that it was conducted continuously and iteratively; and third, that the boundaries between analysis of needs, solution description, and construction were blurred. We describe each of these characteristics in the following.
Solution Description Teams
The solution description team included people who participated in the construction and people with relevant business skills from the customer organization. One informant described their role as being” in front of the development team, before they start Sprint planning, we make a functional solution description with design of screens, error handling, services, conceptual and logical data model” (functional architect). Membership could also span contractors: “If it was only Steria that was involved, then it [solution description] was a business architect, the business representative, and some representatives from the Steria construction teams. If the work spanned contractors, we would involve people from Accenture, Steria, or the Pension Fund” (technical architect). In addition, other people believed to be particularly knowledgeable for the work to be planned could be invited.
At least one member from the feature team (usually the functional architect) would participate in the solution description team. Solution description work was spread among construction team members: “It wasn't one person's responsibility the whole time. It was natural for one to do solution description for a while and then to change it to another from the same team to ensure that everybody knew what we were working on” (technical architect). This practice of involvement reduced the need for overly detailed solution descriptions. Another benefit was pointed out: “It gave good continuity in the work and helped ensure that the construction was smooth and easy” (functional architect).
This was seen as an effective practice for the program regarding knowledge sharing and collaboration, and was taken into account during iteration planning. However, it was challenging for the feature teams because they had to give up team members for a long period—often the most skilled and often in the last phase of the iteration.
A Continuous and Iterative Process
Solution description was a continuous process in two dimensions. It was continuous inside one release, and it was going on in several releases at the same time. Solution description for release n would start while construction for release n-1 was still underway (Fig. 5).
We learned that solution description work in Perform was characterized as “feeding the machine”. The metaphor effectively conveys the property of the construction project as a machine that requires raw material (user stories) to produce products (construct the software system). The main purpose of solution description in Perform was to ensure that the product backlog contained enough high priority work to fill one to three iterations for the feature teams. Furthermore, it was essential to give the machine sufficient raw materials to avoid idle time. Most of the development project members were allocated full-time, and therefore, the cost of running the project “engine” would be constant, whether the backlog was empty or filled.
The detailing level in solution description was subject to change during the project. As the feature teams gained experience, they understood more about the problem domain and less detail was required to convey the essential parts of the tasks. We learned that solution description work became more efficient because “they [the feature teams] understood what we wanted” (functional architect). Informants expressed that it is much easier to communicate effectively when you know the audience. Furthermore, solution description was an iterative activity based on a pragmatic ambition to “detail as much as time permits” (functional architect). Hence, the level of detail was limited by the time available.
Tensions arose from working with a pipeline of completed solution descriptions with as few as one to three iterations. There were noticeable differences between the three suppliers Accenture, Steria, and the Pension Fund in terms of preferences for up-front planning. Accenture had a preference for more detailed, up-front specifications. Accenture wanted reduced risk: “Regarding Pension Fund, we had to ‘discipline’ the customer somewhat, to clarify what they wanted before we took it into our list of committed work, so that we could get the least amount of rework” (functional architect). Steria, on the other hand, was comfortable with more open specifications and continuous collaboration to resolve the details of the software in construction. The Pension Fund had more up-front descriptions than Steria, but less than Accenture.
Blurred Boundaries between Phases
Each release consisted of four consecutive phases: analysis of needs, solution description, construction, and approval. “We could talk to anyone who might have the answer” (technical architect). Customer representatives could approach the feature teams if they needed to discuss or clarify some issues. As a result, the boundaries were blurred between high-level requirements, solution description, and construction: “In practice, high-level requirements and solution description merged more and more" (another technical architect). This enabled more efficient collaboration. The same technical architect stated, "As the contractor starts to understand more about the context, the customer's real problem is more visible, and this means we can find a solution together. Fast.” Blurring the boundaries also gave another interesting result. It became possible for people working on solution description to question the output from the high-level requirement work. A contractor could challenge the Pension Fund: “But could you do it this way instead? Why do it like this?” (technical architect). Project participants found that this way of working stimulated creativity.
The iterative style of detailing the solution descriptions also contributed to blurring the boundaries. For example, the acceptance criteria were supposed to be defined in the solution description phase but were in practice often defined during construction by the teams. Our respondents were unable to agree on what phase the definition of acceptance criteria belonged to. Sometimes, the solution descriptions were too vague to enable construction and were passed back for revision.
We found pragmatism in the allocation of tasks to construction teams. We learned that often, the most challenging tasks were given to the Pension Fund feature teams because they had more domain knowledge and a more flexible contract model: “If there were many changes, if there were complicated modules, and we didn't quite know how to design them, then the Pension Fund teams would get these tasks. I experienced the Pension Fund as being more flexible … without the same constraints in relation to budgets and contracts” (functional architect).
Two topics emerged as primary characteristics related to managing work on software architecture: the tension between up-front and emergent architecture, and the demanding role for architects in large-scale agile projects.
In February 2009, a new chief architect entered Perform. At this point in time, the architecture of the new system was described in terms of architectural strategies but with little compliance in the system under construction. The core principle of the initial architecture was “service orientation”, but it resulted in a complex system with many dependencies and high vulnerability in the event of changes. It was during this period that the decision was made to scale up Perform to more teams. The chief architect’s most pressing concern was to create a system architecture that allowed the maximum parallelism of work in the different teams by “avoiding people stepping on each other’s feet”.
To account for this need, the chief architect and his team created a simpler system architecture focused around service modules and a horizontal unified “work surface” on top. The GUI layer—the “work surface”—made use of all the service modules. The architecture also established rules of ownership to specific parts of the database. An important benefit of the architecture was that it helped to separate the work of different teams into different parts of the code. In Perform, we found that the software architecture for the most part allowed the teams to work autonomously and effectively.
Even if this architecture manifested important decisions for the continued work in the teams, there were tensions stemming from having too little up-front work. Our respondents stated wishes for more “proof of concept” experiments before new technologies were adopted. A notable example was that the Flex technology used in the GUI layer had not been properly understood for it to be used effectively and establish sound “application architecture guidelines”. Our respondents argued that the lack of proof of concepts was due to the contract model: “This was partly governed by the payment model and the program's execution. If they had allocated resources for concept trials that possibly would have to be thrown away, or that resources had been allocated for the development of multiple candidate solutions—but resources were not allocated for that.” (subproject manager, from feedback session).
At the end of the program, there was much work in optimizing the GUI layer code to obtain acceptable performance. Performance had been largely neglected for the main part of the programme but received significant attention in the final stages. This triggered some re-work for the architecture team: “When we started to use this technology [Flex], we established an architecture that was fit only for small applications and small solutions. It was not appropriate for us when we had to fill it up with more logic and GUI elements” (functional architect). Also, significant refactoring was done in the object-persistence layerFootnote 9 on top of the relational database management system for performance reasons. These refactoring efforts were packaged into specific user stories for non-functional requirements.
The programme put emphasis on common architectural principles: “There needs to be openness with regard to suggesting architectural ideas from ‘below’ [in the teams], but you must also be able to convey merits of these ideas to convince the people in the appropriate forums so that it ends up as a unified architecture—so that each team does not use their own solutions” (technical architect, feedback session).
The main architectural structure of the new system remained stable throughout the project. Mostly, the subsequent architectural work was local within the service modules.
The Demanding Role of Architects in Very Large-Scale Programme
We learned that the architect role in very large agile projects is demanding because it requires continuous coordination and negotiation between many stakeholders, including representatives from the business side and the developers. The architect must be able to “sell” technically sound designs and technologies that may have short-term negative impacts on project progress and costs. Another tension was the approach to decision-making in the architecture project. For the most agile-oriented participants, this top-down decision-model came into conflict with their own ambitions to satisfy the business representatives. Team members pointed to a tendency of project architecture to step away from the discussions with business representatives when development teams pointed out that a particular architectural design had cost consequences for the business representatives. When developers argued to the business representatives that a proposed architectural approach would have penalties in terms of cost or performance, subproject architecture responded that it was unfair of developers “to hide behind the business people” instead of complying with architectural principles.
Architecture work was increasingly more difficult for project architecture as well. In the last half of the second year, the pressure to deliver functionality increased: “At this point, project management and other management was concerned about progress … It became much more difficult to promote improvements [in the architecture]” (technical architect). “Yes, more difficult. And then came these issues regarding technical debt, you know? Issues that are horrible in an agile setting” (technical architect). Over time, technical debt would accumulate because the pressure to deliver functionality was so high. Due to the strict schedule in the programme, the business needs were often given priority to long-term architecture. Functionality was given priority over refactoring. We learned that team architects took steps ‘on their own’ to integrate refactoring into normal work tasks: “For me it was equally important to establish the right attitude in the teams regarding code quality as it was to make the business side allocate resources for refactoring” (technical architect, feedback session). We see that architectural work in agile projects is more continuous, and requires more integrative collaboration with a wider range of people.
The two suppliers Accenture and Steria used different approaches to architecture work during the project. Accenture made many architectural decisions during the solution description phase, while Steria delayed many such decisions until the construction. Accenture’s motive was to ensure the highest possible efficiency of the teams’ work during the Iteration. The culture in Steria appeared more flexible and collaborative, seeking rather to clarify issues as they emerged.
Arenas for Inter-Team Coordination
From our data material on coordination and knowledge-sharing between feature teams in the development programme, we identify three main findings. First, there were a number of arenas involved in order to achieve coordination and knowledge sharing. Second, the use of these arenas changed over time, and third, many emphasize the importance of the informal coordination arenas, which were enabled by the open work area. We describe results from each of these three areas in the following.
A Number of Coordination Arenas
Each feature team followed Scrum practices like having a planning meeting, daily stand-up meetings, retrospectives, and demos for each three-week iteration. There were a number of arenas in use to coordinate work and share knowledge between the teams, as shown in Table 4. This list includes mainly formal arenas such as the Scrum of Scrums and Metascrum at a level above the Scrum of Scrum. There were also several informal arenas such as experience forums, lunch seminars, and instant messaging. Some of the arenas were only used within a subproject, such as the “experience forum” at Accenture and the “technical corner” at Steria.
Changing Arenas Over Time
A characteristic of the Perform programme was that arenas for coordination changed over time. "There were meetings that came and disappeared and some that remained" (functional architect). There were information needs that the programme saw was not covered, and new arenas such as the instant messaging were then established. Also, arenas were abandoned because the programme found that they were no longer useful. Open space is an example of an arena that existed for a while and then was abandoned. Some state that this arena had an effect: "During a phase, I felt it had a mission, in particular to motivate people" (subproject manager). Others were more critical of the practice: " …But in total, I feel that we did not get much out of it. Maybe we got to know each other better. Maybe." (another subproject manager).
Several commented on the change from formal mechanisms to informal mechanisms. Over time there was "more usage of boards, more common coffee breaks, more lunches" (subproject manager). Some arenas changed function over time. Internal meetings within the Accenture subproject developed over time into an arena for sharing technical knowledge. This illustrates that there was a change in both the arenas and in how the arenas functioned over time.
Open Work Area
Beginning in 2009, the project was co-located on an open-plan office floor. The teams were organized around tables, and project management for the whole programme and for the projects were on the same floor (see Fig. 5). Many stated that being in one open work area contributed to efficient coordination and knowledge sharing: "It was the best possible solution that the project managers were at one table, in immediate reach of the development teams" (subproject manager). Another subproject manager stated, "I think being on the same floor is important. That is something I notice now being at [another large development programme], where we are not located on the same floor. It is much harder to know what is going on.".
We learned that many of the decisions made in the project were discussed between relevant stakeholders informally and then officially decided upon in one of the formal arenas, the daily Scrums, Scrum of Scrums, or the Metascrum. Although mainly seen as vital to coordination and knowledge sharing, loud discussions on team tables in the open landscape led to some starting to use earphones in order to avoid noise and to indicate that they were not to be disturbed while working.