Transitioning Towards Continuous Experimentation in a Large Software Product and Service Development Organisation – A Case Study

  • Sezin Gizem YamanEmail author
  • Fabian Fagerholm
  • Myriam Munezero
  • Jürgen Münch
  • Mika Aaltola
  • Christina Palmu
  • Tomi Männistö
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10027)


Context: Companies need capabilities to evaluate the customer value of software-intensive products and services. One way of systematically acquiring data on customer value is running continuous experiments as part of the overall development process. Objective: This paper investigates the first steps of transitioning towards continuous experimentation in a large company, including the challenges faced. Method: We conduct a single-case study using participant observation, interviews, and qualitative analysis of the collected data. Results: Results show that continuous experimentation was well received by the practitioners and practising experimentation helped them to enhance understanding of their product value and user needs. Although the complexities of a large multi-stakeholder business-to-business (B2B) environment presented several challenges such as inaccessible users, it was possible to address impediments and integrate an experiment in an ongoing development project. Conclusion: Developing the capability for continuous experimentation in large organisations is a learning process which can be supported by a systematic introduction approach with the guidance of experts. We gained experience by introducing the approach on a small scale in a large organisation, and one of the major steps for future work is to understand how this can be scaled up to the whole development organisation.


Continuous experimentation Experiment-driven software development Product management Lean startup Customer development Customer involvement Organisational transition Agile software development Case study 


  1. 1.
    Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: Building blocks for continuous experimentation. In: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering, RCoSE 2014, pp. 26–35. ACM, New York (2014)Google Scholar
  2. 2.
    Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: The RIGHT model for Continuous Experimentation. J. Syst. Softw. (2016, in press). doi: 10.1016/j.jss.2016.03.034.
  3. 3.
    Rissanen, O., Münch, J.: Continuous experimentation in the B2B domain: a case study. In: Proceedings of the Second International Workshop on Rapid Continuous Software Engineering, RCoSE 2015, Piscataway, NJ, USA, pp. 12–18. IEEE Press (2015)Google Scholar
  4. 4.
    Holmström Olsson, H., Bosch, J.: The HYPEX model: from opinions to data-driven software development. In: Bosch, J. (ed.) Continuous Software Engineering, pp. 155–164. Springer, Cham (2014)Google Scholar
  5. 5.
    Boehm, B., Huang, L.G.: Value-based software engineering: a case study. Computer 36(3), 33–41 (2003)CrossRefGoogle Scholar
  6. 6.
    Highsmith, J., Cockburn, A.: Agile software development: the business of innovation. Computer 34(9), 120–127 (2001)CrossRefGoogle Scholar
  7. 7.
    Cockburn, A., Highsmith, J.: Agile software development, the people factor. Computer 34(11), 131–133 (2001)CrossRefGoogle Scholar
  8. 8.
    Ries, E.: The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business, Houston (2011)Google Scholar
  9. 9.
    Blank, S.: The Four Steps to the Epiphany: Successful Strategies for Products That Win, 2nd edn. K&S Ranch, Pescadero (2013)Google Scholar
  10. 10.
    Croll, A., Yoskowitz, B.: Lean Analytics: Use Data to Build a Better Startup Faster. O’Reilly Media, Sebastopol (2013)Google Scholar
  11. 11.
    Tichy, M., Bosch, J., Goedicke, M., Fitzgerald, B.: 2nd International Workshop on Rapid Continuous Software Engineering (RCoSE 2015). In: Proceedings of the 37th International Conference on Software Engineering, vol. 2, pp. 993–994. IEEE Press (2015)Google Scholar
  12. 12.
    Holmström Olsson, H., Alahyari, H., Bosch, J.: Climbing the “Stairway to Heaven” - a mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In: 2012 38th Euromicro Conference on Software Engineering and Advanced Applications, pp. 392–399 (2012)Google Scholar
  13. 13.
    Yaman, S.G., Sauvola, T., Riungu-Kalliosaari, L., Hokkanen, L., Kuvaja, P., Oivo, M., Männistö, T.: Customer involvement in continuous deployment: a systematic literature review. In: Daneva, M., Pastor, O. (eds.) REFSQ 2016. LNCS, vol. 9619, pp. 249–265. Springer, Heidelberg (2016). doi: 10.1007/978-3-319-30282-9_18 Google Scholar
  14. 14.
    Bosch, J.: Building products as innovation experiment systems. In: Cusumano, M.A., Iyer, B., Venkatraman, N. (eds.) ICSOB 2012. LNBIP, vol. 114, pp. 27–39. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-30746-1_3 CrossRefGoogle Scholar
  15. 15.
    Yin, R.: Case Study Research: Design and Methods, 4th edn. SAGE Publications, Inc., Thousand Oaks (2009)Google Scholar
  16. 16.
    Robson, C.: Real World Research. Wiley, Chichester (2011)Google Scholar
  17. 17.
    Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3(2), 77–101 (2006)CrossRefGoogle Scholar
  18. 18.
    North, D.: Introducing BDD. Better Software., March 2006Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Sezin Gizem Yaman
    • 1
    Email author
  • Fabian Fagerholm
    • 1
  • Myriam Munezero
    • 1
  • Jürgen Münch
    • 1
    • 2
  • Mika Aaltola
    • 3
  • Christina Palmu
    • 3
  • Tomi Männistö
    • 1
  1. 1.Department of Computer ScienceUniversity of HelsinkiHelsinkiFinland
  2. 2.Reutlingen UniversityBöblingenGermany
  3. 3.EricssonJorvasFinland

Personalised recommendations