Prioritizing agile benefits and limitations in relation to practice usage

Abstract

In recent years, there has been significant shift from rigid development (RD) toward agile. However, it has also been spotted that agile methodologies are hardly ever followed in their pure form. Hybrid processes as combinations of RD and agile practices emerge. In addition, agile adoption has been reported to result in both benefits and limitations. This exploratory study (a) identifies development models based on RD and agile practice usage by practitioners; (b) identifies agile practice adoption scenarios based on eliciting practice usage over time; (c) prioritizes agile benefits and limitations in relation to (a) and (b). Practitioners provided answers through a questionnaire. The development models are determined using hierarchical cluster analysis. The use of practices over time is captured through an interactive board with practices and time indication sliders. This study uses the extended hierarchical voting analysis framework to investigate benefit and limitation prioritization. Four types of development models and six adoption scenarios have been identified. Overall, 45 practitioners participated in the prioritization study. A common benefit among all models and adoption patterns is knowledge and learning, while high requirements on professional skills were perceived as the main limitation. Furthermore, significant variances in terms of benefits and limitations have been observed between models and adoption patterns. The most significant internal benefit categories from adopting agile are knowledge and learning, employee satisfaction, social skill development, and feedback and confidence. Professional skill-specific demands, scalability, and lack of suitability for specific product domains are the main limitations of agile practice usage. Having a balanced agile process allows to achieve a high number of benefits. With respect to adoption, a big bang transition from RD to agile leads to poor quality in comparison with the alternatives.

Appendices

Appendix 1: Surveyed practices

The following practices have been surveyed:

• 1. Up-front documentation of requirements: Requirements are fully documented and as complete as possible

• 2. Detailed up-front architecture design: Detailed and extensive design of entire software architecture, documentation-heavy

• 3. Big bang integration and infrequent releases: Most of the implementation artifacts, such as code and modules, are integrated at once before milestones; releases are done at least once per 4 months or more

• 4. Extensive time planning: Fully specified Gantt charts with fixed milestones for the entire project, complete work breakdown structure decomposition,

• 5. The project follows a sequential flow of phases: When one phase is finished, it is closed, e.g., (0) requirements, (1) design, (2) construction, (3) testing, (4) delivery.

• 6. Detailed management plans and process documentation: Formal and plan-based quality, risk, resource, change, and configuration management

• 7. Repeatable development process: The process is formally defined and followed, it is repeatable, its flexibility and abilities to adapt to projects are limited

• 8. Face-to-face communication: Team sits together, open space office facilitating interaction, video conferences if the team is distributed

• 9. Technical excellence: Ongoing refactoring of the code, simplest solution design, following coding standards

• 10. Small self-organizing cross-functional teams: The team is independent, takes full responsibility; small with no more than 10 members

• 11. On-site customer: Continuous user involvement in the development process, customer can be consulted anytime if it is needed

• 12. Frequent planning/reporting: Everyday meetings to discuss what was and what is going to be done, reporting the progress on a burn-down chart, burn-up chart, informative workshops

• 13. Prioritized list of requirements: Tasks in a backlog as features/user stories—short statements of the functionality, system metaphors—stories about how the system works

• 14. Pair-programming: Two developers work together at one workstation, they switch ro

• 15. Time-boxing: Fixed time of iterations, meetings, working hours limited to 40 h weekly—employees hardly ever work overtime, no “death march” projects, team members avoid burnout, etc.

• 16. Continuous integration with testing: Software is built frequently, even few times a day, accompanied with testing (e.g., ten-minute builds, automated unit, regression, etc.)

• 17. Short iterations and releases: Frequent releases of the software, at most 3–4 months, early and continuous delivery of partial but fully functional software

• 18. Iteration reviews, retrospectives: The entire team participates in selecting features to be implemented in the following iteration, estimating resources required to implement them, consensus based, e.g., planning game, the Wideband Delphi Estimation, planning poker, etc.

• 19. Iteration reviews, retrospectives: Meetings after each iteration to review the project, discuss threats to process efficiency, modify, and improve, build up the software development process

• 20. Test-driven development: Writing automated test cases for functionalities and then implementing the tested functionalities until the tests are passed successfully

• 21. Collective code ownership: Everybody in the team can change the code of other developers in case of maintenance, bug fixing or other development activities

Appendix 2: Overview of internal benefits derived from literature

The following list shows the benefit categories surveyed (categories 1 to 10) and the items within the categories.

• 1. Knowledge and Learning: Agile results in improved knowledge transfer and helps to achieve the teams a common understanding.

  • 1.1. Agile improves knowledge transfer and learning between team members (e.g., how to do good testing, how to write good user stories, domain knowledge) as communication is improved (e.g., frequent contacts, exchange of programmers) (Pikkarainen et al. 2012; Petersen 2011; Petersen and Wohlin 2009; Vijayasarathy and Turk 2008).

  • 1.2. Agile helps teams to achieve a common understanding (e.g., shared vision of the product requirements and the development process to be used) among team members (Pikkarainen et al. 2012; Laanti et al. 2011; Dybå and Dingsøyr 2008; Petersen and Wohlin 2009; Begel and Nagappan 2007).

• 2. Social Skills Development: Agile helps to develop socials skills such as communication, cooperation, adaptability.

  • 2.1. Agile helps to develop communication skills (e.g., presentation, constructive discussion, and negotiations) (Dybå and Dingsøyr 2008).

  • 2.2. Agile helps to develop cooperation-related skills (e.g., ability to help each other and solve problems together) (Dybå and Dingsøyr 2008; Petersen and Wohlin 2009).

  • 2.3. Agile helps to develop adaptability skills (e.g., ability to adapt to changing conditions/change in plans) (Dybå and Dingsøyr 2008).

• 3. Employee Satisfaction: Agile improves the quality of working life. Employees feel more comfortable, relaxed, satisfied, and purposeful.

  • 3.1. Agile leads to empowerment of the engineers and thus increased motivation due to its technical focus (e.g., frequent milestones and short iterations) (Petersen and Wohlin 2009).

  • 3.2. Agile makes people feel comfortable and relaxed, yet purposeful and productive (e.g. due to common ownership of product, 40 h week, discussion of issues) (Dybå and Dingsøyr 2008; Petersen and Wohlin 2009).

  • 3.3. Agile increases confidence and satisfaction (from the perspective of the internal organization) of the product being developed as it gives the confidence that the developed software is what the customer desires (e.g., the team is capable to achieve at hand and quickly understand where the limitations were) (Pikkarainen et al. 2012; Dybå and Dingsøyr 2008; Petersen and Wohlin 2009; Vijayasarathy and Turk 2008).

  • 3.4. Agile makes people feel purposeful as the team is a self-organizing community with a culture that emphasizes shared responsibility (e.g., team members have influence on decisions, there is a need to collaborate in order to achieve goals) (Dybå and Dingsøyr 2008).

• 4. Feedback and confidence: Agile increases the confidence of development team that what they are doing is appropriate. The feedback from customer is frequent and decisions are verified very soon.

  • 4.1. Agile allows to verify design decisions very soon due to frequent feedback (e.g., feedback from integration tests) (Petersen and Wohlin 2009).

  • 4.2. Agile increases the awareness of what has to be developed thanks to close cooperation with the customer (e.g., by on-site customers) (Dybå and Dingsøyr 2008; Petersen 2011; Petersen and Wohlin 2010).

  • 4.3. Agile leads to soon requirements validation due to frequent feedback on their work (e.g., feedback from customers after software presentations) (Begel and Nagappan 2007; Vijayasarathy and Turk 2008).

• 5. Quality: Agile results in improved quality of software products.

  • 5.1. Agile results in higher quality and reuse of code (e.g., ongoing refactoring, test-first programming, and programming in pairs) (Laanti et al. 2011; Begel and Nagappan 2007; Petersen and Wohlin 2009; Vijayasarathy and Turk 2008).

  • 5.2. Agile leads to improved quality of design, architecture and performance as design decisions are verified very soon (Begel and Nagappan 2007).

• 6. Verification and validation processes improvement: Agile leads to improved, more efficient verification and validation processes.

  • 6.1. Agile leads to improved turnaround time for fixing bugs due to early discovery and handling of issues and thus following development relies on verified work (Begel and Nagappan 2007; Petersen 2011).

  • 6.2. Agile provides much quicker and more thorough response to defects as responsibility for measuring quality is moved from the managers to the developers (Dybå and Dingsøyr 2008).

  • 6.3. Agile leads to more efficient use of testers’ time as in small teams testing and design can be easily parallelized due to short ways of communication between designers and testers (instant testing) (Petersen and Wohlin 2009).

• 7. Planning and estimations: Agile allows for more accurate planning and estimations based on real data.

  • 7.1. Agile allows accurate small scope estimations as requirements in requirements packages are precise (Petersen and Wohlin 2009).

  • 7.2. Agile allows more efficient way of project planning as the processes are organized in more releases, and managers are more satisfied with the way they plan their projects (rather than are RD companies) (Dybå and Dingsøyr 2008).

  • 7.3. Agile increases the ability to forecast (e.g., releases and iterations) based on real data as the process is more dynamic (Begel and Nagappan 2007; Vijayasarathy and Turk 2008).

• 8. Monitoring and controlling: Agile results in more efficient ways of monitoring and controlling of the development process.

  • 8.1. Agile methods complement the stage-gate management model with powerful tools for microplanning, day-to-day work control, reporting on progress (Dybå and Dingsøyr 2008).

  • 8.2. Agile methods result in improved resource management, product functionality, on-time delivery, and quality control due to real-time project tracking (Dybå and Dingsøyr 2008).

  • 8.3. Agile leads to increased process control, transparency, and quality through continuous integration and small manageable tasks (Begel and Nagappan 2007; Petersen and Wohlin 2009).

  • 8.4. Agile allows for continuous improvement and refinement of the development process (e.g., at the end of each iteration and release) (Begel and Nagappan 2007; Petersen and Wohlin 2009).

  • 8.5. Agile allows to fail cheaply as each fail is detected on every small step (e.g., end of iteration and daily meetings) (Begel and Nagappan 2007; Vijayasarathy and Turk 2008).

• 9. Adaptability: Agile is adaptable to different environments as well as other software development approaches.

  • 9.1. Agile thrives in radically different environments (e.g., from hierarchical structure to little or no central control, customer involvement, and physical setting varied greatly) (Dybå and Dingsøyr 2008).

  • 9.2. Agile methods are adaptable, and it is possible to combine agile project management with overall traditional principles (e.g., RD approaches) (Dybå and Dingsøyr 2008).

• 10. Decreased effort: Agile leads to much more efficient development process due to reduction in effort of work, design, etc.

  • 10.1. Agile decreases effort related to changing requirements as the process is flexible to incorporate changes in requirements at a later stage with less impact on the project (e.g., features and requirements that are planned in detail are only these to be implemented in a specific cycle) (Dybå and Dingsøyr 2008).

  • 10.2. Agile helps to decrease effort for redesign due to very soon architectural design decisions verification (e.g., frequent feedback and software demos) since the later changes are incorporated, the more effort is generated (Petersen and Wohlin 2009).

  • 10.3. Agile decreases effort of overall work due to reduction in the waste of not used work (requirements documented, components implemented) as it is reduced as small packages started are always implemented (Laanti et al. 2011; Petersen and Wohlin 2009).

  • 10.4. Agile decreases effort of rework caused by faults as testing priorities are made more clear due to prioritized features, and that testers as well as designers work closely together (Petersen and Wohlin 2009).

  • 10.5. Agile decreases the effort of extensive documenting since small teams with people having different roles only require small amounts of documentation as it is replaced with direct communication facilitating learning and understanding for each other (Petersen and Wohlin 2009).

Appendix 3: Overview of limitations derived from the literature

The following list shows the limitation categories surveyed (categories 11 to 17) and the items within the categories.

• 11. Employee Dissatisfaction: Agile results in decreased motivation and increased stress level of the employees.

  • 11.1. Agile decreases developers motivation as they are distracted and feel uncomfortable when they have to report progress of work (e.g., work that has not been tested in an integrated fashion) as a matter of frequent meetings (Begel and Nagappan 2007).

  • 11.2. Agile causes increase in stress level of developers due to frequent deadlines and milestones (Petersen and Wohlin 2009).

• 12. Professional Skill-specific Demands: Agile is very demanding when considering professional skills in order to be successful.

  • 12.1. Agile requires a comparable level of qualifications of the developers, otherwise the development will be ineffective (e.g., delays in development of modules) (Dybå and Dingsøyr 2008; Petersen and Wohlin 2009).

  • 12.2. Agile requires high qualifications from the all team members along with the leader to be effective (e.g., poorly conducted meetings and lack of knowledge of prioritization techniques) (Pikkarainen et al. 2012; Begel and Nagappan 2007; Petersen and Wohlin 2009, 2010).

  • 12.3. Agile requires sufficiently trained managers (on agile development) as the empowerment of engineers makes managers afraid initially (the teams should be self-organized rather than centrally managed) (Pikkarainen et al. 2012; Begel and Nagappan 2007; Petersen and Wohlin 2009; Vijayasarathy and Turk 2008).

  • 12.4. Agile makes team members less interchangeable (what has consequences for how projects are managed) and more difficult to describe and identify (agile teams are not only build of skilled developers, members complement each other, synergy is created) (Dybå and Dingsøyr 2008; Begel and Nagappan 2007).

• 13. Limitation to Specific Product Domains: Agile is limited to safety-critical and legacy systems.

  • 13.1. Agile provides limited support for developing safety-critical software (e.g., military or health care domain where the software needs to be of the highest quality possible). Petersen and Wohlin (2010).

  • 13.2. Agile provides limited support for the development of legacy systems Begel and Nagappan (2007).

• 14. Scalability: Agile is poorly scalable and not applicable to large development teams.

  • 14.1. Agile is poorly scalable and thus provides limited support for distributed development environments Petersen and Wohlin (2010).

  • 14.2. Agile is not applicable to large teams and thus limited for development large-scale development Pikkarainen et al. (2012); Laanti et al. (2011); Begel and Nagappan (2007); Petersen and Wohlin (2009, 2010); Vijayasarathy and Turk (2008).

• 15. Increased Testing Effort: Agile results in increased testing effort. Continuous testing demands are very high.

  • 15.1. Agile requires much effort to compensate shortage of project testing due to test coverage reduction within projects as a result of lack of independent testing Pikkarainen et al. (2012); Petersen and Wohlin (2009).

  • 15.2. Agile requires much effort to produce big amounts of testing documentation Petersen and Wohlin (2009).

  • 15.3. Agile requires much effort for continuous testing as creating an integrated test environment is hard for different platforms and system dependencies Petersen and Wohlin (2009, 2010).

• 16. Increased Process Effort: Agile methods lead to increased effort of project management, software maintenance, packaging, etc.

  • 16.1. Agile significantly increases the effort of configuration management due to the need of coordination of the high number of internal releases Petersen and Wohlin (2009).

  • 16.2. Agile methods cause increase in effort of packaging as it is still viewed from a technical point of view, but not from a commercial point of view Petersen and Wohlin (2009).

  • 16.3. Agile increases the development effort and delays as dependencies rooted in implementation details are hard to identify and not covered in the anatomy plan (lack of documentation) Petersen and Wohlin (2009).

  • 16.4. Agile increases the management overhead due to a high number of teams requiring much coordination and communication between (issues with coordination of work, priority lists between agile and non-agile teams due to different approaches, phases, methodology, etc.) Petersen and Wohlin (2009).

  • 16.5. Agile increases the maintenance effort as many different versions have to be supported and test environments for different versions have to be created (e.g., as result of the ability of releasing many releases to market) Petersen and Wohlin (2009).

• 17. Lead Time and Efficiency: Agile is often inefficient. The inefficiency is mainly related to development, communication overhead, and delays.

  • 17.1. Agile is inefficient as there is a handover from requirements to design due to complex decision processes as well as some decisions are ineffective due to social values embraced by agile as they are contrary to those that group members desire Pikkarainen et al. (2012); Petersen and Wohlin (2009).

  • 17.2. Agile results in delays of getting functional testing running due to workload of frequent integration Petersen and Wohlin (2009).

  • 17.3. Agile provides inefficient way of reporting progress on frequent meetings (e.g., if team is inexperienced or more than 8–10 people) Begel and Nagappan (2007).

  • 17.4. Agile results in reduced efficiency of development as close developers cooperation leads to exhaustion (Dybå and Dingsøyr 2008; Petersen and Wohlin 2009).

  • 17.5. Agile makes the inter-team communication inefficient (e.g., especially when teams follow different software development process and have to cooperate within the same project) (Begel and Nagappan 2007).