Empirical Software Engineering

, Volume 20, Issue 5, pp 1384–1425 | Cite as

On rapid releases and software testing: a case study and a semi-systematic literature review

  • Mika V. Mäntylä
  • Bram Adams
  • Foutse Khomh
  • Emelie Engström
  • Kai Petersen


Large open and closed source organizations like Google, Facebook and Mozilla are migrating their products towards rapid releases. While this allows faster time-to-market and user feedback, it also implies less time for testing and bug fixing. Since initial research results indeed show that rapid releases fix proportionally less reported bugs than traditional releases, this paper investigates the changes in software testing effort after moving to rapid releases in the context of a case study on Mozilla Firefox, and performs a semi-systematic literature review. The case study analyzes the results of 312,502 execution runs of the 1,547 mostly manual system-level test cases of Mozilla Firefox from 2006 to 2012 (5 major traditional and 9 major rapid releases), and triangulates our findings with a Mozilla QA engineer. We find that rapid releases have a narrower test scope that enables a deeper investigation of the features and regressions with the highest risk. Furthermore, rapid releases make testing more continuous and have proportionally smaller spikes before the main release. However, rapid releases make it more difficult to build a large testing community , and they decrease test suite diversity and make testing more deadline oriented. In addition, our semi-systematic literature review presents the benefits, problems and enablers of rapid releases from 24 papers found using systematic search queries and a similar amount of papers found through other means. The literature review shows that rapid releases are a prevalent industrial practice that are utilized even in some highly critical domains of software engineering, and that rapid releases originated from several software development methodologies such as agile, open source, lean and internet-speed software development. However, empirical studies proving evidence of the claimed advantages and disadvantages of rapid releases are scarce.


Software testing Release model Builds Bugs Open-source Agile releases Mozilla 



We would like to thank the Mozilla QA engineer who provided feedback to our findings. His responses are accounts of personal experience and opinion, and are in no means whatsoever an official statement from Mozilla. This work has been partially supported by EU FP7 Grant 318082 (U-QASAR,, ELLIIT (the StrategicArea for ICT research, funded by the Swedish Government,, N4S (research program managed by DIGILE and funded by TEKES,, and NSERC (Natural Sciences and Engineering Research Council of Canada).


  1. Agarwal P (2011) Continuous scrum: Agile management of saas products, pp 51–60Google Scholar
  2. Azham Z, Ghani I, Ithnin N (2011) Security backlog in scrum security practices, pp 414–417Google Scholar
  3. Baskerville R, Ramesh B, Levine L, Pries-Heje J (2006) High-speed software development practices: What works, what doesn’t. IT Professional 8:29–36CrossRefGoogle Scholar
  4. Baysal O, Davis I, Godfrey MW (2011) A tale of two browsers. In: Proceedings of the 8th Working Conf. on Mining Software Repositories (MSR), pp 238–241Google Scholar
  5. Baysal O, Kononenko O, Holmes R, Godfrey M (2012) The secret life of patches: a firefox case study, pp 447–455Google Scholar
  6. Beck K, Andres C (2004) Extreme Programming explained: embrace change, 2nd edn. Addison-WesleyGoogle Scholar
  7. Begel A, Nagappan N (2007) Usage and perceptions of agile software development in an industrial context: an exploratory study, pp 255–264Google Scholar
  8. Bell R, Ostrand T, Weyuker E (2006) Looking for bugs in all the right places, vol. 2006, pp 61–71Google Scholar
  9. Berner S, Weber R, Keller RK (2005) Observations and lessons learned from automated testing. In: Proceedings of the 27th international conference on Software engineering, ACM, pp 571–579Google Scholar
  10. Boshuizen C, Marshall W, Bridges C, Kenyon S, Klupar P (2011) Learning to follow: embracing commercial technologies and open source for space missions 10:8097–8101Google Scholar
  11. Brown AW (2011) A case study in agile-at-scale delivery. In: Proceeding of the 12th Intl. Conf. on Agile Processes in Software Engineering and Extreme Programming (XP), vol 77, pp 266–281Google Scholar
  12. Byrt T, Bishop J, Carlin JB (1993) Bias, prevalence and kappa. J Clin Epidemiol 46:423–429CrossRefGoogle Scholar
  13. Carney PA, Sickles EA, Monsees BS, Bassett LW, Brenner RJ, Feig SA, Smith RA, Rosenberg RD, Bogart TA, Browning S et al (2010) Identifying minimally acceptable interpretive performance criteria for screening mammography 1. Radiology 255:354–361CrossRefGoogle Scholar
  14. Cicchetti DV, Feinstein AR (1990) High agreement but low kappa: Ii. resolving the paradoxes. J Clin Epidemiol 43:551–558CrossRefGoogle Scholar
  15. Codabux Z, Williams B (2013) Managing technical debt: an industrial case study, pp 8–15Google Scholar
  16. Cohan S (2007) Successful integration of agile development techniques within disa, pp 255–260Google Scholar
  17. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Academic PressGoogle Scholar
  18. Cusumano MA, Yoffie DB (1999) Competing on internet time: Lessons from netscape and its battle with microsoft. Simon and SchusterGoogle Scholar
  19. Gamma E (2005) Agile, open source, distributed, and on-time – inside the eclipse development process. Keynote at the 27th Intl. Conf. on Software Engineering (ICSE)Google Scholar
  20. Greiler M, van Deursen A, Zaidman A (2012) Measuring test case similarity to support test suite understanding. In: Objects, models components patterns. Springer, pp 91–107Google Scholar
  21. Gundebahar M, Kus Khalilov M (2013) A hybrid deployment model for financial systems with service oriented architecture (soa): Running from client via branch server, pp 365–370Google Scholar
  22. Heikkilä V, Rautiainen K, Jansen S (2010) A revelatory case study on scaling agile release planning, pp 289–296Google Scholar
  23. Hemmati H, Arcuri A, Briand L (2013) Achieving scalable model-based testing through test case diversity. ACM Trans Softw Eng Methodol (TOSEM) 22:6CrossRefGoogle Scholar
  24. Hodgetts P, Phillips D (2002) 30. In: eXtreme Adoption eXperiences of a B2B Start Up. Addison-Wesley Longman Publishing Co., Inc. Extreme Programming PerspectivesGoogle Scholar
  25. Hollander M, Wolfe DA (1999) Nonparametric statistical methods, 2nd edn. John Wiley and Sons incGoogle Scholar
  26. HP (2012) Shorten release cycles aby bringing developers to application lifecycle management. HP applications handbook, Retrieved on February 08, 2012Google Scholar
  27. Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation, 1st edn. Addison-Wesley ProfessionalGoogle Scholar
  28. InvestmentWatch (2011) Mozilla puts out firefox 5.0 web browser which carries over 1,000 improvements in just about 3 months of development.
  29. Iwai A, Aoyama M (2011) Automotive cloud service systems based on service-oriented architecture and its evaluation, pp 638–645Google Scholar
  30. Jansen S, Brinkkemper S (2006) Ten misconceptions about product software release management explained using update cost/value functions. In: Proceedings of the International Workshop on Software Product Management, pp 44–50Google Scholar
  31. Jenkins J (2011) Velocity culture (the unmet challenge in ops). Presentation at O’Reilly Velocity ConferenceGoogle Scholar
  32. Jones G, Leung V (2005) Visual surveillance: a systems engineering approach for rapid development. Number 2005-11033, pp 161–166Google Scholar
  33. Kaply M (2012) Why do companies stay on old technology? Retrieved on January 12, 2012Google Scholar
  34. Kettunen V, Kasurinen J, Taipale O, Smolander K (2010) A study on agility and testing processes in software organizations. In: Proc. of the 19th Intl. Symp. on Software Testing and Analysis (ISSTA), pp 231–240Google Scholar
  35. Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? An empirical case study of mozilla firefox. In: MSR, pp 179–188Google Scholar
  36. Kim S (2012) ppcor: Partial and semi-partial (part) correlation.
  37. Kitchenham B (2004) Procedures for performing systematic reviews, vol. 33:2004Google Scholar
  38. Kong S, Kendall JE, Kendall KE (2009) The challenge of improving software quality: Developers’ beliefs about the contribution of agile practices. In: Proc. of the Americas Conf. on Information Systems (AMCIS), p 12Google Scholar
  39. Kühner G, Bluhm T, Heimann P, Hennig C, Kroiss H, Krom J, Laqua H, Lewerentz M, Maier J, Schacht J, Spring A, Werner A, Zilker M (2012) Progress on standardization and automation in software development on w7x. Fusion Eng Des 87:2232–2237CrossRefGoogle Scholar
  40. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics159–174Google Scholar
  41. Lavoie T, Merlo E (2013) How much really changes? a case study of firefox version evolution using a clone detector, pp 83–89Google Scholar
  42. Li J, Moe NB, Dybå T (2010) Transition from a plan-driven process to scrum: a longitudinal case study on software quality. In: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp 13:1–13:10Google Scholar
  43. Li X, Li Y, Xie M, Ng S (2011) Reliability analysis and optimal version-updating for open source software. Inf Softw Technol 53:929–936CrossRefGoogle Scholar
  44. Ltd. RS (2013) Web browsers (global marketshare).
  45. Maalej W (2009) Task-first or context-first? tool integration revisited, pp 344–355Google Scholar
  46. Mäntylä MV, Itkonen J (2013) More testers âĂŞ the effect of crowd size and time restriction in software testing. Inf Softw Technol 55:986–1003CrossRefGoogle Scholar
  47. Mantyla M, Khomh F, Adams B, Engstrom E, Petersen K (2013) On rapid releases and software testing. In: Proceedings of the 29th IEEE International Conference on Software Maintenance (ICSM), pp 20–29Google Scholar
  48. Marschall M (2007) Transforming a six month release cycle to continuous flow. In: Proc. of the conf. on AGILE, pp 395–400Google Scholar
  49. Martin D, Rooksby J, Rouncefield M, Sommerville I (2007) ’good’organisational reasons for’bad’software testing: An ethnographic study of testing in a small software company. In: Software Engineering, 2007. ICSE 2007. 29th International Conference on, IEEE, pp 602–611Google Scholar
  50. McDaniel LS (1990) The effects of time pressure and audit program structure on audit performance. J Account Res 28:267–285CrossRefGoogle Scholar
  51. Middleton P (2001) Lean software development two case studies. Software Qual J 9:241–252CrossRefGoogle Scholar
  52. Middleton P, Joyce D (2012) Lean software management: Bbc worldwide case study. IEEE Trans Eng Manag 59:20–32CrossRefGoogle Scholar
  53. Mozilla (2013) Litmus wiki.
  54. Mozilla (2013) Moztrap wiki.
  55. Mozilla (2013) Mozilla source code mercurial repositoriesGoogle Scholar
  56. Olsson H, Alahyari H, Bosch J (2012) Climbing the “stairway to heaven” - a mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software, pp 392–399Google Scholar
  57. Olsson H, Bosch J, Alahyari H (2013) Towards r&d as innovation experiment systems: a framework for moving beyond agile software development, pp 798–805Google Scholar
  58. Otte T, Moreton R, Knoell HD (2008) Applied quality assurance methods under the open source development model. In: Proceeding of the 32nd Annual IEEE Intl. Computer Software and Applications Conf. (COMPSAC), pp 1247–1252Google Scholar
  59. Oza N, Ebert C, Abrahamsson P (2012) Lean software development. IEEE Softw 29:0022–25Google Scholar
  60. Paasivaara M, Lassenius C (2003) Collaboration practices in global inter-organizational software development projects. Softw Process Improv Pract 8:183–199CrossRefGoogle Scholar
  61. Patel C, Lycett M, Macredie R, De Cesare S (2006) Perceptions of agility and collaboration in software development practice, vol. 1, p 10cGoogle Scholar
  62. Paul R (2011) Mozilla outlines 16-week firefox development cycle.
  63. Paul R (2012) Firefox extended support will mitigate rapid release challenges.
  64. Pérez-Castillo R, Sánchez-González L, Piattini M, García F, Garcia-Rodriguez de Guzman I (2011) Obtaining thresholds for the effectiveness of business process mining. In: Empirical software engineering and measurement (ESEM), 2011 International Symposium on IEEE, pp 453–462Google Scholar
  65. Petersen K, Wohlin C (2009) A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case. J Syst Softw 82:1479–1490CrossRefGoogle Scholar
  66. Petersen K, Wohlin C (2010) The effect of moving from a plan-driven to an incremental software development approach with agile practices. Empir Softw Eng 15:654–693CrossRefGoogle Scholar
  67. Poppendieck M (2007) Lean software development. In: Companion to the proceedings of the 29th International Conference on Software Engineering, IEEE Computer Society, pp 165–166Google Scholar
  68. Porter A, Yilmaz C, Memon AM, Krishna AS, Schmidt DC, Gokhale A (2006) Techniques and processes for improving the quality and performance of open-source software. Softw Process Improv Pract 11:163–176CrossRefGoogle Scholar
  69. Rafi DM, Moses KRK, Petersen K, Mäntylä MV (2012) Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. In: Automation of Software Test (AST), 2012 7th International Workshop on, IEEE, pp 36–42Google Scholar
  70. Raymond ES (1999) The Cathedral and the Bazaar, 1st edn. O’Reilly & Associates, Inc., SebastopolGoogle Scholar
  71. Reinertsen DG (2009) The principles of product development flow: second generation lean product development. Celeritas Redondo Beach, CanadaGoogle Scholar
  72. Rogmann JJ (2013) Orddom: Ordinal dominance statistics.
  73. Ruhe G, Saliu MO (2005) The art and science of software release planning. IEEE Softw 22:47–53CrossRefGoogle Scholar
  74. Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14:131–164CrossRefGoogle Scholar
  75. Schwaber K (1995) Scrum development process. In: Proceedings of the 10th Annual ACM Conference on Object Oriented Programming Systems, Languages, and Applications (OOPSLA), pp 117–134Google Scholar
  76. Shankland S (2010) Google ethos speeds up chrome release cycle.
  77. Shankland S (2011) Rapid-release firefox meets corporate backlash.
  78. Sicore D (2011) New channels for firefox rapid releases.
  79. Sonnentag S, Brodbeck FC, Heinbokel T, Stolte W (1994) Stressor-burnout relationship in software development teams. J Occup Organ Psychol 67:327–341CrossRefGoogle Scholar
  80. Sundmark D, Petersen K, Larsson S (2011) An exploratory case study of testing in an automotive electrical system release process, pp 166–175Google Scholar
  81. Szőke Á (2011) Conceptual scheduling model and optimized release scheduling for agile environments. Inf Softw Technol 53:574–591CrossRefGoogle Scholar
  82. Torkar R, Minoves P, Garrigós J (2011) Adopting free/libre/open source software practices, techniques and methods for industrial use. J Assoc Inf Syst 12:88–122Google Scholar
  83. Wikipedia (2013) Firefox release history.
  84. Wikipedia (2013) Extended support release.
  85. Wikipedia (2013) Locale.
  86. VersionOne (2012) 7th annual state of agile survey.

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Mika V. Mäntylä
    • 1
  • Bram Adams
    • 2
  • Foutse Khomh
    • 3
  • Emelie Engström
    • 4
  • Kai Petersen
    • 5
  1. 1.Department of Computer Science and EngineeringAalto UniversityEspooFinland
  2. 2.MCISPolytechnique MontréalQuébecCanada
  3. 3.SWATPolytechnique MontréalQuébecCanada
  4. 4.Department of Computer ScienceLund UniversityLundSweden
  5. 5.School of ComputingBlekinge Institute of TechnologyKarlskronaSweden

Personalised recommendations